Squashed commit of the following:

commit 12bb37ef0a430ab30d0ec3e06d7f58d42b1332b9
Author: Dylan <dylanpdx@gmail.com>
Date:   Mon Sep 12 17:31:22 2022 +0100

    Increase memory to 1GB

commit bbdd0f85c500413f765b44cdb8643539a01b3968
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 23:19:41 2022 +0100

    Reduced memory to 256mb

commit 5fa80151a09092a7206ffd294ef464650b776ba8
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 23:10:57 2022 +0100

    Updated serverless code

commit 42f826908ac82a91e67312ae6270ce6ac9bfd216
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 19:33:39 2022 +0100

    Squashed commit of the following:

    commit 27accd7a5a601bd77a3c13832e92ebbd0571b7b5
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 23:54:15 2022 +0100

        Documented apiMirrors

    commit bf46166b027133719ebbd5d1ed46dcae8d242190
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:58:53 2022 +0100

        Fixed incorrect log print 😅

    commit a05ff596e057c774a77fa20381370a1baa60cf20
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:52:20 2022 +0100

        Fix #14

    commit 1954a72bd070d213f539c8a8260b6de9c98a0a59
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:33:07 2022 +0100

        API Mirror

    commit c58e9b27554d00c4545a70f62a9b5a7c132563d5
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 21:38:42 2022 +0100

        Updated requirements

    commit a51876a82943567107ef9c777cfa5ffd9c89f060
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jul 23 16:34:36 2022 +0100

        Fix twitter api failing for direct video linking

    commit d97f70b22494899417cc5fc8744e1d4ea6d22f9e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jul 23 16:29:39 2022 +0100

        Added option to disable cache

    commit 2da2cd52005ac7e7a2827a56cfb894b2294a769c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri Jul 22 00:57:07 2022 +0100

        Removed image scale-down at the end of processing

    commit b0547ce8107979bac0b32946490f1a4e578cb8fb
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jul 17 14:00:19 2022 +0100

        Same fix as last commit

    commit 1ec59bce2ea8a6543854fe24e4eeb738f4499794
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jul 17 13:54:59 2022 +0100

        Fixed direct linking on Discord

    commit 2a77ff96b05dabc916ad4b516897d68477c5d41f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jul 6 19:43:22 2022 +0100

        Separated code into different files

    commit 973c31f705f8e5f86416e3607c8c75fe6c0cc1c1
    Merge: e28789a b27296e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 9 14:35:33 2022 +0100

        Merge pull request #7 from flifloo/main

        Add Docker image and compose with documentation

    commit b27296e317e0eec87245fba2a58a38fdcb09e8ec
    Author: flifloo <flifloo@gmail.com>
    Date:   Tue Jun 7 11:46:39 2022 +0200

        Add Docker image and compose with documentation

    commit 40f4770e177e4003f12758b9f75c3373e56562c2
    Author: flifloo <flifloo@gmail.com>
    Date:   Tue Jun 7 10:44:52 2022 +0200

        Fix links.json creation

    commit e28789a09dbb6efd36827a3fd47e0132666a9609
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jun 5 13:36:18 2022 +0100

        Added og:description on videos for Telegram users

    commit faed7482f4b3d09a5656bf5d2a078018bd223e94
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jun 5 02:10:54 2022 +0100

        Update requirements.txt

    commit a5b2eb6cd0472d83a8b0f37ddfa2b358cb5d31f4
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jun 4 13:53:14 2022 +0100

        Restore old behavior of showing error message

    commit 16dfc509d32fe6667a43f9008daf9031bf93a1c4
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri Jun 3 23:27:33 2022 +0100

        Small log message when an unofficial API call succeeds

    commit 174a20d8961728f45041de00ae1b8e3c30fb2d5d
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 2 17:07:48 2022 +0100

        Use unofficial API when official API fails

    commit e64dee93498620b0c95b305d72178f23e506dfb6
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 2 01:09:04 2022 +0100

        Temporary redirect for failing links due to API ratelimiting

    commit f0f709f62ec40dd3138bde5cff7b027f82d82eb6
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 21:52:47 2022 +0100

        use yt-dlp

    commit 0f77b2e0a466a09dabbc4eff33b4ce8517fc8148
    Merge: 285c8f7 ffcde09
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 19:26:22 2022 +0000

        Merge branch 'main' of github.com:dylanpdx/BetterTwitFix

    commit 285c8f70f84d5efe96d50a9e239c7e460be5cd38
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 19:25:23 2022 +0000

        Service auto-restart

    commit ffcde09b976dd85630da73cb4c804b86c63fb760
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 30 01:25:53 2022 +0100

        Return HTTP 400 if missing query parameters

    commit 891db049af0de4d40df3eaabe67d12952a036224
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri May 27 23:18:58 2022 +0100

        Fixed incorrect TTL being set

    commit 797b670d0e968075cb98de37b1439c59c4470caa
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 18:11:30 2022 +0100

        Do not combine single image tweets

    commit 728785278b549d467e51e6f8fb4492bc5db71cf9
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:48:24 2022 +0100

        Fixed direct video linking

    commit ec1f4dc2845956e4237610e29ebad65b544ea095
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:13:31 2022 +0100

        Image check in lambda handler

    commit 77b972a1b2069e039c5067eb7dd2e95bcb2acea1
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:10:29 2022 +0100

        Switched to using ThreadPoolExecutor

    commit 4feb1b65d6ec0fcff2854b7080f8995e6ae77fe1
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 15:01:12 2022 +0100

        Optimized image downloads

    commit 2819b504a36f7b39d61640b87cb0a81e804e36a7
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 02:01:43 2022 +0100

        Add quality argument

    commit 2399788fdc0d8a10063f30c448462e9caa79997f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 01:29:47 2022 +0100

        Misc code cleanup; Fixed Discord not embedding 301 redirects issue

    commit 7a021916f0c9e8404646104258394e764080bffc
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 00:58:28 2022 +0100

        Use proper mime type

    commit a06a89825cf29d07787286d4837dfcb7a0151760
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:56:48 2022 +0100

        Fixed formatting

    commit 6717fd464aaa934724ea9c9f6022402013c7238b
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:55:41 2022 +0100

        Documented combination_method

    commit 022edb5122ddef8a8b3f84ffe84b1b2e4cf8130f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:48:55 2022 +0100

        Moved combineImg to own module; created AWS dockerfile; added combination_method config

    commit d7e0cb90898ac1aacfbe600f6dd1d8137185791a
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 17:28:03 2022 +0100

        Changed scaling method

    commit 55ea554357aca026926e1d296fb5ac259458c1d3
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 16:30:51 2022 +0100

        Re-scale image down

    commit b457760fae648c3cad5e383717f4df28c4539303
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 01:20:27 2022 +0100

        Use JPEG for final combined img

    commit 39a9ffd5120f7d28711e071290ee00d495a477ef
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 00:49:22 2022 +0100

        Method to embed combined tweet images. WIP & may be CPU intensive

    commit a6efd26447efa9b54e02c48bda9b58b038236ae0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 23 20:33:11 2022 +0100

        Oops- Updated other two error msgs

    commit e81beb975ce319ccc95fa8db9e3e39927e205929
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 23 19:02:54 2022 +0100

        Improved Fail link message

    commit 55c956ad4ec7fe0ef0cbfb0e740bfcb1783aa88f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 22 15:19:28 2022 +0100

        Error checking for loading cache json file

    commit 014f9d0e8d94da562f834665741adda744c0b42c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 22 15:19:17 2022 +0100

        TTL for cache entries

    commit f640ac69a1ba7f19ad69f6fea2199bec696d5eaf
    Merge: 7eff745 08d686c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat May 21 16:13:26 2022 +0100

        Merge pull request #2 from cabiste69/patch-1

        Minor cleaning

    commit 08d686c652a5edbf8252771c30a53743b5541f33
    Author: cabiste <ahmedjj354@gmail.com>
    Date:   Fri May 20 23:14:58 2022 +0100

        minor cleaning

    commit 7eff745550631e2d0ad014de9e941151bb5a8b61
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 18 18:23:11 2022 +0100

        Clarifications in readme

    commit 8df211090c4d73c812b76dba65c6dc180c848300
    Merge: 5a2a4e6 681f191
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Tue May 17 17:14:30 2022 +0100

        Merge pull request #1 from Fam0r/main

        Use the best bitrate video for twitter embeds

    commit 5a2a4e6316dfd1369b7a428ed4c9fcb9fe794de5
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Tue May 17 14:55:49 2022 +0100

        Display how many total images in the post

    commit 681f19150219c22a22999ac1e41dc5d393d5b9b5
    Author: Fam0r <fam0r@mailbox.org>
    Date:   Tue May 17 09:40:40 2022 +0300

        Use the best bitrate video for twitter embeds

    commit 82fe66af48c5dc4f8d933f2874f8bae815e7dd05
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 16 20:18:18 2022 +0100

        Work in progress code for combining up to 4 images

    commit fba1256efd3ce539f716e2f23b45c122c8422759
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:42:22 2022 +0100

        Fixed infinite redirect(?)

    commit 85c682ab111f575c8c517117f456cf354e9f5bf7
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:15:15 2022 +0100

        Updated gitignore

commit f2eefbe5a542f1a6fa8404228c2a9425054765a8
Author: Dylan <dylanpdx@gmail.com>
Date:   Wed May 18 18:21:00 2022 +0100

    Fixed DynamoDB integration

commit 32e5376b987c732147ab3289c2c42a8afde19e2f
Author: Dylan <dylanpdx@gmail.com>
Date:   Mon May 16 19:28:31 2022 +0100

    DynamoDB support & Serverless config update

commit dee62f812a6c8da443b111c7f3dd24bcb4d76525
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 17:06:42 2022 +0100

    Updated serverless.yml to dockerize pip packages

commit 43d69a1d872c293a4b70d2a330261245783eb924
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 14:30:18 2022 +0100

    serverless wsgi files

commit 96246aa9217774ddc88275ffba5bf1790317eda9
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 01:47:20 2022 +0100

    Squashed commit of the following:

    commit fba1256efd3ce539f716e2f23b45c122c8422759
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:42:22 2022 +0100

        Fixed infinite redirect(?)

    commit 85c682ab111f575c8c517117f456cf354e9f5bf7
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:15:15 2022 +0100

        Updated gitignore

commit 2c9563fafe97bfe7782386aaa04372dcac5f8136
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 01:14:41 2022 +0100

    Serverless config
This commit is contained in:
Dylan 2022-09-12 17:36:11 +01:00
parent 27accd7a5a
commit 10ec46a0cf
6 changed files with 10970 additions and 34 deletions

View File

@ -3,9 +3,14 @@ import pymongo
from datetime import date,datetime
import json
import os
import boto3
link_cache_system = config['config']['link_cache']
DYNAMO_CACHE_TBL=None
if link_cache_system=="dynamodb":
DYNAMO_CACHE_TBL=os.environ['CACHE_TABLE']
if link_cache_system == "json":
link_cache = {}
if not os.path.exists("links.json"):
@ -28,6 +33,8 @@ elif link_cache_system == "db":
client = pymongo.MongoClient(config['config']['database'], connect=False)
table = config['config']['table']
db = client[table]
elif link_cache_system == "dynamodb":
client = boto3.resource('dynamodb')
def serializeUnknown(obj):
if isinstance(obj, (datetime, date)):
@ -45,8 +52,21 @@ def addVnfToLinkCache(video_link, vnf):
with open("links.json", "w") as outfile:
json.dump(link_cache, outfile, indent=4, sort_keys=True, default=serializeUnknown)
return None
except Exception:
elif link_cache_system == "dynamodb":
vnf["ttl"] = int(vnf["ttl"].strftime('%s'))
table = client.Table(DYNAMO_CACHE_TBL)
table.put_item(
Item={
'tweet': video_link,
'vnf': vnf,
'ttl':vnf["ttl"]
}
)
print(" ➤ [ + ] Link added to dynamodb cache ")
return True
except Exception as e:
print(" ➤ [ X ] Failed to add link to DB cache")
print(e)
return None
def getVnfFromLinkCache(video_link):
@ -72,5 +92,19 @@ def getVnfFromLinkCache(video_link):
else:
print(" ➤ [ X ] Link not in json cache")
return None
elif link_cache_system == "dynamodb":
table = client.Table(DYNAMO_CACHE_TBL)
response = table.get_item(
Key={
'tweet': video_link
}
)
if 'Item' in response:
print("Link located in dynamodb cache")
vnf = response['Item']['vnf']
return vnf
else:
print(" ➤ [ X ] Link not in dynamodb cache")
return None
elif link_cache_system == "none":
return None

View File

@ -1,33 +1,54 @@
import json
import os
# Read config from config.json. If it does not exist, create new.
if not os.path.exists("config.json"):
with open("config.json", "w") as outfile:
default_config = {
if ('RUNNING_SERVERLESS' in os.environ and os.environ['RUNNING_SERVERLESS'] == '1'):
config = {
"config":{
"link_cache":"json",
"database":"[url to mongo database goes here]",
"table":"TwiFix",
"method":"youtube-dl",
"color":"#43B581",
"appname": "vxTwitter",
"repo": "https://github.com/dylanpdx/BetterTwitFix",
"url": "https://vxtwitter.com",
"combination_method": "local" # can either be 'local' or a URL to a server handling requests in the same format
"link_cache":os.environ["VXTWITTER_LINK_CACHE"],
"database":os.environ["VXTWITTER_DATABASE"],
"table":os.environ["VXTWITTER_DATABASE_TABLE"],
"method":os.environ["VXTWITTER_METHOD"],
"color":os.environ["VXTWITTER_COLOR"],
"appname": os.environ["VXTWITTER_APP_NAME"],
"repo": os.environ["VXTWITTER_REPO"],
"url": os.environ["VXTWITTER_URL"],
"combination_method": os.environ["VXTWITTER_COMBINATION_METHOD"] # can either be 'local' or a URL to a server handling requests in the same format
},
"api":{"api_key":"[api_key goes here]",
"api_secret":"[api_secret goes here]",
"access_token":"[access_token goes here]",
"access_secret":"[access_secret goes here]",
"api":{"api_key":os.environ["VXTWITTER_TWITTER_API_KEY"],
"api_secret":os.environ["VXTWITTER_TWITTER_API_SECRET"],
"access_token":os.environ["VXTWITTER_TWITTER_ACCESS_TOKEN"],
"access_secret":os.environ["VXTWITTER_TWITTER_ACCESS_SECRET"],
"apiMirrors":[]
}
}
json.dump(default_config, outfile, indent=4, sort_keys=True)
config = default_config
else:
f = open("config.json")
config = json.load(f)
f.close()
# Read config from config.json. If it does not exist, create new.
if not os.path.exists("config.json"):
with open("config.json", "w") as outfile:
default_config = {
"config":{
"link_cache":"json",
"database":"[url to mongo database goes here]",
"table":"TwiFix",
"method":"youtube-dl",
"color":"#43B581",
"appname": "vxTwitter",
"repo": "https://github.com/dylanpdx/BetterTwitFix",
"url": "https://vxtwitter.com",
"combination_method": "local" # can either be 'local' or a URL to a server handling requests in the same format
},
"api":{"api_key":"[api_key goes here]",
"api_secret":"[api_secret goes here]",
"access_token":"[access_token goes here]",
"access_secret":"[access_secret goes here]",
"apiMirrors":[]
}
}
json.dump(default_config, outfile, indent=4, sort_keys=True)
config = default_config
else:
f = open("config.json")
config = json.load(f)
f.close()

10771
package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

28
package.json Normal file
View File

@ -0,0 +1,28 @@
{
"name": "bettertwitfix",
"version": "1.0.0",
"description": "(A fork of TwitFix)\r Basic flask server that serves fixed twitter video embeds to desktop discord by using either the Twitter API or Youtube-DL to grab tweet video information. This also automatically embeds the first link in the text of non video tweets (API Only)",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "git+https://github.com/dylanpdx/BetterTwitFix.git"
},
"keywords": [],
"author": "",
"license": "ISC",
"bugs": {
"url": "https://github.com/dylanpdx/BetterTwitFix/issues"
},
"homepage": "https://github.com/dylanpdx/BetterTwitFix#readme",
"dependencies": {
"serverless-wsgi": "^3.0.0"
},
"devDependencies": {
"serverless-plugin-common-excludes": "^4.0.0",
"serverless-plugin-include-dependencies": "^5.0.0",
"serverless-python-requirements": "^5.4.0"
}
}

83
serverless.yml Normal file
View File

@ -0,0 +1,83 @@
service: vxTwitter
provider:
name: aws
runtime: python3.8
stage: dev
iamRoleStatements:
- Effect: Allow
Action:
- dynamodb:Query
- dynamodb:Scan
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
Resource:
- { "Fn::GetAtt": ["vxTwitterDynamoTable", "Arn" ] }
environment:
CACHE_TABLE: ${self:custom.tableName}
RUNNING_SERVERLESS: 1
VXTWITTER_LINK_CACHE: dynamodb
VXTWITTER_DATABASE: none
VXTWITTER_DATABASE_TABLE: none
VXTWITTER_METHOD: hybrid
VXTWITTER_COLOR: \#43B581
VXTWITTER_APP_NAME: vxTwitter
VXTWITTER_REPO: https://github.com/dylanpdx/BetterTwitFix
VXTWITTER_URL: https://vxtwitter.com
VXTWITTER_COMBINATION_METHOD: local
# Twitter API keys
VXTWITTER_TWITTER_API_KEY: none
VXTWITTER_TWITTER_API_SECRET: none
VXTWITTER_TWITTER_ACCESS_TOKEN: none
VXTWITTER_TWITTER_ACCESS_SECRET: none
package:
patterns:
- '!node_modules/**'
- '!venv/**'
plugins:
- serverless-wsgi
- serverless-python-requirements
- serverless-plugin-common-excludes
- serverless-plugin-include-dependencies
functions:
vxTwitterApp:
handler: wsgi_handler.handler
url: true
timeout: 15
MemorySize: 1024
layers:
- Ref: PythonRequirementsLambdaLayer
custom:
tableName: 'tweets-table-${self:provider.stage}'
wsgi:
app: twitfix.app
pythonRequirements:
layer: true
dockerizePip: true
resources:
Resources:
vxTwitterDynamoTable:
Type: 'AWS::DynamoDB::Table'
Properties:
AttributeDefinitions:
-
AttributeName: tweet
AttributeType: S
KeySchema:
-
AttributeName: tweet
KeyType: HASH
TableName: ${self:custom.tableName}
BillingMode: PAY_PER_REQUEST
TimeToLiveSpecification:
AttributeName: ttl
Enabled: true

View File

@ -38,7 +38,6 @@ generate_embed_user_agents = [
"test"]
# If method is set to API or Hybrid, attempt to auth with the Twitter API
if config['config']['method'] in ('api', 'hybrid'):
auth = twitter.oauth.OAuth(config['api']['access_token'], config['api']['access_secret'], config['api']['api_key'], config['api']['api_secret'])
@ -371,17 +370,17 @@ def link_to_vnf_from_youtubedl(video_link):
def link_to_vnf(video_link): # Return a VideoInfo object or die trying
if config['config']['method'] == 'hybrid':
#try:
# return link_to_vnf_from_api(video_link)
#except Exception as e:
# print(" ➤ [ !!! ] API Failed")
#print(e)
try:
return link_to_vnf_from_api(video_link)
return link_to_vnf_from_unofficial_api(video_link)
except Exception as e:
print(" ➤ [ !!! ] API Failed")
print(" ➤ [ !!! ] UNOFFICIAL API Failed")
print(e)
try:
return link_to_vnf_from_unofficial_api(video_link)
except Exception as e:
print(" ➤ [ !!! ] UNOFFICIAL API Failed")
print(e)
return link_to_vnf_from_youtubedl(video_link) # This is the last resort, will only work for videos
return link_to_vnf_from_youtubedl(video_link) # This is the last resort, will only work for videos
elif config['config']['method'] == 'api':
try: