Squashed commit of the following:

commit 12bb37ef0a430ab30d0ec3e06d7f58d42b1332b9
Author: Dylan <dylanpdx@gmail.com>
Date:   Mon Sep 12 17:31:22 2022 +0100

    Increase memory to 1GB

commit bbdd0f85c500413f765b44cdb8643539a01b3968
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 23:19:41 2022 +0100

    Reduced memory to 256mb

commit 5fa80151a09092a7206ffd294ef464650b776ba8
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 23:10:57 2022 +0100

    Updated serverless code

commit 42f826908ac82a91e67312ae6270ce6ac9bfd216
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 19:33:39 2022 +0100

    Squashed commit of the following:

    commit 27accd7a5a
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 23:54:15 2022 +0100

        Documented apiMirrors

    commit bf46166b02
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:58:53 2022 +0100

        Fixed incorrect log print 😅

    commit a05ff596e0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:52:20 2022 +0100

        Fix #14

    commit 1954a72bd0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:33:07 2022 +0100

        API Mirror

    commit c58e9b2755
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 21:38:42 2022 +0100

        Updated requirements

    commit a51876a829
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jul 23 16:34:36 2022 +0100

        Fix twitter api failing for direct video linking

    commit d97f70b224
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jul 23 16:29:39 2022 +0100

        Added option to disable cache

    commit 2da2cd5200
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri Jul 22 00:57:07 2022 +0100

        Removed image scale-down at the end of processing

    commit b0547ce810
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jul 17 14:00:19 2022 +0100

        Same fix as last commit

    commit 1ec59bce2e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jul 17 13:54:59 2022 +0100

        Fixed direct linking on Discord

    commit 2a77ff96b0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jul 6 19:43:22 2022 +0100

        Separated code into different files

    commit 973c31f705
    Merge: e28789a b27296e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 9 14:35:33 2022 +0100

        Merge pull request #7 from flifloo/main

        Add Docker image and compose with documentation

    commit b27296e317
    Author: flifloo <flifloo@gmail.com>
    Date:   Tue Jun 7 11:46:39 2022 +0200

        Add Docker image and compose with documentation

    commit 40f4770e17
    Author: flifloo <flifloo@gmail.com>
    Date:   Tue Jun 7 10:44:52 2022 +0200

        Fix links.json creation

    commit e28789a09d
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jun 5 13:36:18 2022 +0100

        Added og:description on videos for Telegram users

    commit faed7482f4
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jun 5 02:10:54 2022 +0100

        Update requirements.txt

    commit a5b2eb6cd0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jun 4 13:53:14 2022 +0100

        Restore old behavior of showing error message

    commit 16dfc509d3
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri Jun 3 23:27:33 2022 +0100

        Small log message when an unofficial API call succeeds

    commit 174a20d896
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 2 17:07:48 2022 +0100

        Use unofficial API when official API fails

    commit e64dee9349
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 2 01:09:04 2022 +0100

        Temporary redirect for failing links due to API ratelimiting

    commit f0f709f62e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 21:52:47 2022 +0100

        use yt-dlp

    commit 0f77b2e0a4
    Merge: 285c8f7 ffcde09
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 19:26:22 2022 +0000

        Merge branch 'main' of github.com:dylanpdx/BetterTwitFix

    commit 285c8f70f8
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 19:25:23 2022 +0000

        Service auto-restart

    commit ffcde09b97
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 30 01:25:53 2022 +0100

        Return HTTP 400 if missing query parameters

    commit 891db049af
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri May 27 23:18:58 2022 +0100

        Fixed incorrect TTL being set

    commit 797b670d0e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 18:11:30 2022 +0100

        Do not combine single image tweets

    commit 728785278b
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:48:24 2022 +0100

        Fixed direct video linking

    commit ec1f4dc284
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:13:31 2022 +0100

        Image check in lambda handler

    commit 77b972a1b2
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:10:29 2022 +0100

        Switched to using ThreadPoolExecutor

    commit 4feb1b65d6
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 15:01:12 2022 +0100

        Optimized image downloads

    commit 2819b504a3
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 02:01:43 2022 +0100

        Add quality argument

    commit 2399788fdc
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 01:29:47 2022 +0100

        Misc code cleanup; Fixed Discord not embedding 301 redirects issue

    commit 7a021916f0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 00:58:28 2022 +0100

        Use proper mime type

    commit a06a89825c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:56:48 2022 +0100

        Fixed formatting

    commit 6717fd464a
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:55:41 2022 +0100

        Documented combination_method

    commit 022edb5122
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:48:55 2022 +0100

        Moved combineImg to own module; created AWS dockerfile; added combination_method config

    commit d7e0cb9089
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 17:28:03 2022 +0100

        Changed scaling method

    commit 55ea554357
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 16:30:51 2022 +0100

        Re-scale image down

    commit b457760fae
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 01:20:27 2022 +0100

        Use JPEG for final combined img

    commit 39a9ffd512
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 00:49:22 2022 +0100

        Method to embed combined tweet images. WIP & may be CPU intensive

    commit a6efd26447
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 23 20:33:11 2022 +0100

        Oops- Updated other two error msgs

    commit e81beb975c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 23 19:02:54 2022 +0100

        Improved Fail link message

    commit 55c956ad4e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 22 15:19:28 2022 +0100

        Error checking for loading cache json file

    commit 014f9d0e8d
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 22 15:19:17 2022 +0100

        TTL for cache entries

    commit f640ac69a1
    Merge: 7eff745 08d686c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat May 21 16:13:26 2022 +0100

        Merge pull request #2 from cabiste69/patch-1

        Minor cleaning

    commit 08d686c652
    Author: cabiste <ahmedjj354@gmail.com>
    Date:   Fri May 20 23:14:58 2022 +0100

        minor cleaning

    commit 7eff745550
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 18 18:23:11 2022 +0100

        Clarifications in readme

    commit 8df211090c
    Merge: 5a2a4e6 681f191
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Tue May 17 17:14:30 2022 +0100

        Merge pull request #1 from Fam0r/main

        Use the best bitrate video for twitter embeds

    commit 5a2a4e6316
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Tue May 17 14:55:49 2022 +0100

        Display how many total images in the post

    commit 681f191502
    Author: Fam0r <fam0r@mailbox.org>
    Date:   Tue May 17 09:40:40 2022 +0300

        Use the best bitrate video for twitter embeds

    commit 82fe66af48
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 16 20:18:18 2022 +0100

        Work in progress code for combining up to 4 images

    commit fba1256efd
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:42:22 2022 +0100

        Fixed infinite redirect(?)

    commit 85c682ab11
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:15:15 2022 +0100

        Updated gitignore

commit f2eefbe5a542f1a6fa8404228c2a9425054765a8
Author: Dylan <dylanpdx@gmail.com>
Date:   Wed May 18 18:21:00 2022 +0100

    Fixed DynamoDB integration

commit 32e5376b987c732147ab3289c2c42a8afde19e2f
Author: Dylan <dylanpdx@gmail.com>
Date:   Mon May 16 19:28:31 2022 +0100

    DynamoDB support & Serverless config update

commit dee62f812a6c8da443b111c7f3dd24bcb4d76525
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 17:06:42 2022 +0100

    Updated serverless.yml to dockerize pip packages

commit 43d69a1d872c293a4b70d2a330261245783eb924
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 14:30:18 2022 +0100

    serverless wsgi files

commit 96246aa9217774ddc88275ffba5bf1790317eda9
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 01:47:20 2022 +0100

    Squashed commit of the following:

    commit fba1256efd
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:42:22 2022 +0100

        Fixed infinite redirect(?)

    commit 85c682ab11
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:15:15 2022 +0100

        Updated gitignore

commit 2c9563fafe97bfe7782386aaa04372dcac5f8136
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 01:14:41 2022 +0100

    Serverless config

cache.py

@@ -3,9 +3,14 @@ import pymongo
 from datetime import date,datetime
 import json
 import os
+import boto3
 
 link_cache_system = config['config']['link_cache']
 
+DYNAMO_CACHE_TBL=None
+if link_cache_system=="dynamodb":
+    DYNAMO_CACHE_TBL=os.environ['CACHE_TABLE']
+
 if link_cache_system == "json":
     link_cache = {}
     if not os.path.exists("links.json"):
@@ -28,6 +33,8 @@ elif link_cache_system == "db":
     client = pymongo.MongoClient(config['config']['database'], connect=False)
     table = config['config']['table']
     db = client[table]
+elif link_cache_system == "dynamodb":
+    client = boto3.resource('dynamodb')
 
 def serializeUnknown(obj):
     if isinstance(obj, (datetime, date)):
@@ -45,8 +52,21 @@ def addVnfToLinkCache(video_link, vnf):
             with open("links.json", "w") as outfile: 
                 json.dump(link_cache, outfile, indent=4, sort_keys=True, default=serializeUnknown)
                 return None
-    except Exception:
+        elif link_cache_system == "dynamodb":
+            vnf["ttl"] = int(vnf["ttl"].strftime('%s'))
+            table = client.Table(DYNAMO_CACHE_TBL)
+            table.put_item(
+                Item={
+                    'tweet': video_link,
+                    'vnf': vnf,
+                    'ttl':vnf["ttl"]
+                }
+            )
+            print(" ➤ [ + ] Link added to dynamodb cache ")
+            return True
+    except Exception as e:
         print(" ➤ [ X ] Failed to add link to DB cache")
+        print(e)
         return None
 
 def getVnfFromLinkCache(video_link):
@@ -72,5 +92,19 @@ def getVnfFromLinkCache(video_link):
         else:
             print(" ➤ [ X ] Link not in json cache")
             return None
+    elif link_cache_system == "dynamodb":
+        table = client.Table(DYNAMO_CACHE_TBL)
+        response = table.get_item(
+            Key={
+                'tweet': video_link
+            }
+        )
+        if 'Item' in response:
+            print("Link located in dynamodb cache")
+            vnf = response['Item']['vnf']
+            return vnf
+        else:
+            print(" ➤ [ X ] Link not in dynamodb cache")
+            return None
     elif link_cache_system == "none":
         return None

configHandler.py

@@ -1,8 +1,29 @@
 import json
 import os
 
-# Read config from config.json. If it does not exist, create new.
+if ('RUNNING_SERVERLESS' in os.environ and os.environ['RUNNING_SERVERLESS'] == '1'):
-if not os.path.exists("config.json"):
+    config = {
+            "config":{
+                "link_cache":os.environ["VXTWITTER_LINK_CACHE"],
+                "database":os.environ["VXTWITTER_DATABASE"],
+                "table":os.environ["VXTWITTER_DATABASE_TABLE"],
+                "method":os.environ["VXTWITTER_METHOD"], 
+                "color":os.environ["VXTWITTER_COLOR"], 
+                "appname": os.environ["VXTWITTER_APP_NAME"], 
+                "repo": os.environ["VXTWITTER_REPO"], 
+                "url": os.environ["VXTWITTER_URL"],
+                "combination_method": os.environ["VXTWITTER_COMBINATION_METHOD"] # can either be 'local' or a URL to a server handling requests in the same format
+                },
+            "api":{"api_key":os.environ["VXTWITTER_TWITTER_API_KEY"],
+            "api_secret":os.environ["VXTWITTER_TWITTER_API_SECRET"],
+            "access_token":os.environ["VXTWITTER_TWITTER_ACCESS_TOKEN"],
+            "access_secret":os.environ["VXTWITTER_TWITTER_ACCESS_SECRET"],
+            "apiMirrors":[]
+            }
+        }
+else:
+    # Read config from config.json. If it does not exist, create new.
+    if not os.path.exists("config.json"):
         with open("config.json", "w") as outfile:
             default_config = {
                 "config":{
@@ -27,7 +48,7 @@ if not os.path.exists("config.json"):
             json.dump(default_config, outfile, indent=4, sort_keys=True)
 
         config = default_config
-else:
+    else:
         f = open("config.json")
         config = json.load(f)
         f.close()

package.json

@@ -0,0 +1,28 @@
+{
+  "name": "bettertwitfix",
+  "version": "1.0.0",
+  "description": "(A fork of TwitFix)\r Basic flask server that serves fixed twitter video embeds to desktop discord by using either the Twitter API or Youtube-DL to grab tweet video information. This also automatically embeds the first link in the text of non video tweets (API Only)",
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/dylanpdx/BetterTwitFix.git"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "bugs": {
+    "url": "https://github.com/dylanpdx/BetterTwitFix/issues"
+  },
+  "homepage": "https://github.com/dylanpdx/BetterTwitFix#readme",
+  "dependencies": {
+    "serverless-wsgi": "^3.0.0"
+  },
+  "devDependencies": {
+    "serverless-plugin-common-excludes": "^4.0.0",
+    "serverless-plugin-include-dependencies": "^5.0.0",
+    "serverless-python-requirements": "^5.4.0"
+  }
+}

serverless.yml

@@ -0,0 +1,83 @@
+service: vxTwitter
+
+provider:
+  name: aws
+  runtime: python3.8
+  stage: dev
+  iamRoleStatements:
+    - Effect: Allow
+      Action:
+        - dynamodb:Query
+        - dynamodb:Scan
+        - dynamodb:GetItem
+        - dynamodb:PutItem
+        - dynamodb:UpdateItem
+        - dynamodb:DeleteItem
+      Resource:
+        - { "Fn::GetAtt": ["vxTwitterDynamoTable", "Arn" ] }
+  environment:
+    CACHE_TABLE: ${self:custom.tableName}
+    RUNNING_SERVERLESS: 1
+    VXTWITTER_LINK_CACHE: dynamodb
+    VXTWITTER_DATABASE: none
+    VXTWITTER_DATABASE_TABLE: none
+    VXTWITTER_METHOD: hybrid
+    VXTWITTER_COLOR: \#43B581
+    VXTWITTER_APP_NAME: vxTwitter
+    VXTWITTER_REPO: https://github.com/dylanpdx/BetterTwitFix
+    VXTWITTER_URL: https://vxtwitter.com
+    VXTWITTER_COMBINATION_METHOD: local
+    # Twitter API keys
+    VXTWITTER_TWITTER_API_KEY: none
+    VXTWITTER_TWITTER_API_SECRET: none
+    VXTWITTER_TWITTER_ACCESS_TOKEN: none
+    VXTWITTER_TWITTER_ACCESS_SECRET: none
+
+package:
+  patterns:
+    - '!node_modules/**'
+    - '!venv/**'
+
+plugins:
+  - serverless-wsgi
+  - serverless-python-requirements
+  - serverless-plugin-common-excludes
+  - serverless-plugin-include-dependencies
+
+functions:
+  vxTwitterApp:
+    handler: wsgi_handler.handler
+    url: true
+    timeout: 15
+    MemorySize: 1024
+    layers:
+      - Ref: PythonRequirementsLambdaLayer
+
+
+custom:
+  tableName: 'tweets-table-${self:provider.stage}'
+  wsgi:
+    app: twitfix.app
+  pythonRequirements:
+    layer: true
+    dockerizePip: true
+
+	
+resources:
+  Resources:
+    vxTwitterDynamoTable:
+      Type: 'AWS::DynamoDB::Table'
+      Properties:
+        AttributeDefinitions:
+          -
+            AttributeName: tweet
+            AttributeType: S
+        KeySchema:
+          -
+            AttributeName: tweet
+            KeyType: HASH
+        TableName: ${self:custom.tableName}
+        BillingMode: PAY_PER_REQUEST
+        TimeToLiveSpecification:
+          AttributeName: ttl
+          Enabled: true

twitfix.py

@@ -38,7 +38,6 @@ generate_embed_user_agents = [
     "test"]
 
 
-
 # If method is set to API or Hybrid, attempt to auth with the Twitter API
 if config['config']['method'] in ('api', 'hybrid'):
     auth = twitter.oauth.OAuth(config['api']['access_token'], config['api']['access_secret'], config['api']['api_key'], config['api']['api_secret'])
@@ -371,11 +370,11 @@ def link_to_vnf_from_youtubedl(video_link):
 
 def link_to_vnf(video_link): # Return a VideoInfo object or die trying
     if config['config']['method'] == 'hybrid':
-        try:
+        #try:
-            return link_to_vnf_from_api(video_link)
+        #    return link_to_vnf_from_api(video_link)
-        except Exception as e:
+        #except Exception as e:
-            print(" ➤ [ !!! ] API Failed")
+        #    print(" ➤ [ !!! ] API Failed")
-            print(e)
+        #print(e)
         try:
             return link_to_vnf_from_unofficial_api(video_link)
         except Exception as e: