Dylan 10ec46a0cf Squashed commit of the following:
commit 12bb37ef0a430ab30d0ec3e06d7f58d42b1332b9
Author: Dylan <dylanpdx@gmail.com>
Date:   Mon Sep 12 17:31:22 2022 +0100

    Increase memory to 1GB

commit bbdd0f85c500413f765b44cdb8643539a01b3968
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 23:19:41 2022 +0100

    Reduced memory to 256mb

commit 5fa80151a09092a7206ffd294ef464650b776ba8
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 23:10:57 2022 +0100

    Updated serverless code

commit 42f826908ac82a91e67312ae6270ce6ac9bfd216
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Sep 8 19:33:39 2022 +0100

    Squashed commit of the following:

    commit 27accd7a5a601bd77a3c13832e92ebbd0571b7b5
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 23:54:15 2022 +0100

        Documented apiMirrors

    commit bf46166b027133719ebbd5d1ed46dcae8d242190
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:58:53 2022 +0100

        Fixed incorrect log print 😅

    commit a05ff596e057c774a77fa20381370a1baa60cf20
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:52:20 2022 +0100

        Fix #14

    commit 1954a72bd070d213f539c8a8260b6de9c98a0a59
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 22:33:07 2022 +0100

        API Mirror

    commit c58e9b27554d00c4545a70f62a9b5a7c132563d5
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Sep 1 21:38:42 2022 +0100

        Updated requirements

    commit a51876a82943567107ef9c777cfa5ffd9c89f060
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jul 23 16:34:36 2022 +0100

        Fix twitter api failing for direct video linking

    commit d97f70b22494899417cc5fc8744e1d4ea6d22f9e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jul 23 16:29:39 2022 +0100

        Added option to disable cache

    commit 2da2cd52005ac7e7a2827a56cfb894b2294a769c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri Jul 22 00:57:07 2022 +0100

        Removed image scale-down at the end of processing

    commit b0547ce8107979bac0b32946490f1a4e578cb8fb
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jul 17 14:00:19 2022 +0100

        Same fix as last commit

    commit 1ec59bce2ea8a6543854fe24e4eeb738f4499794
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jul 17 13:54:59 2022 +0100

        Fixed direct linking on Discord

    commit 2a77ff96b05dabc916ad4b516897d68477c5d41f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jul 6 19:43:22 2022 +0100

        Separated code into different files

    commit 973c31f705f8e5f86416e3607c8c75fe6c0cc1c1
    Merge: e28789a b27296e
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 9 14:35:33 2022 +0100

        Merge pull request #7 from flifloo/main

        Add Docker image and compose with documentation

    commit b27296e317e0eec87245fba2a58a38fdcb09e8ec
    Author: flifloo <flifloo@gmail.com>
    Date:   Tue Jun 7 11:46:39 2022 +0200

        Add Docker image and compose with documentation

    commit 40f4770e177e4003f12758b9f75c3373e56562c2
    Author: flifloo <flifloo@gmail.com>
    Date:   Tue Jun 7 10:44:52 2022 +0200

        Fix links.json creation

    commit e28789a09dbb6efd36827a3fd47e0132666a9609
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jun 5 13:36:18 2022 +0100

        Added og:description on videos for Telegram users

    commit faed7482f4b3d09a5656bf5d2a078018bd223e94
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun Jun 5 02:10:54 2022 +0100

        Update requirements.txt

    commit a5b2eb6cd0472d83a8b0f37ddfa2b358cb5d31f4
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat Jun 4 13:53:14 2022 +0100

        Restore old behavior of showing error message

    commit 16dfc509d32fe6667a43f9008daf9031bf93a1c4
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri Jun 3 23:27:33 2022 +0100

        Small log message when an unofficial API call succeeds

    commit 174a20d8961728f45041de00ae1b8e3c30fb2d5d
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 2 17:07:48 2022 +0100

        Use unofficial API when official API fails

    commit e64dee93498620b0c95b305d72178f23e506dfb6
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu Jun 2 01:09:04 2022 +0100

        Temporary redirect for failing links due to API ratelimiting

    commit f0f709f62ec40dd3138bde5cff7b027f82d82eb6
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 21:52:47 2022 +0100

        use yt-dlp

    commit 0f77b2e0a466a09dabbc4eff33b4ce8517fc8148
    Merge: 285c8f7 ffcde09
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 19:26:22 2022 +0000

        Merge branch 'main' of github.com:dylanpdx/BetterTwitFix

    commit 285c8f70f84d5efe96d50a9e239c7e460be5cd38
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed Jun 1 19:25:23 2022 +0000

        Service auto-restart

    commit ffcde09b976dd85630da73cb4c804b86c63fb760
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 30 01:25:53 2022 +0100

        Return HTTP 400 if missing query parameters

    commit 891db049af0de4d40df3eaabe67d12952a036224
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Fri May 27 23:18:58 2022 +0100

        Fixed incorrect TTL being set

    commit 797b670d0e968075cb98de37b1439c59c4470caa
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 18:11:30 2022 +0100

        Do not combine single image tweets

    commit 728785278b549d467e51e6f8fb4492bc5db71cf9
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:48:24 2022 +0100

        Fixed direct video linking

    commit ec1f4dc2845956e4237610e29ebad65b544ea095
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:13:31 2022 +0100

        Image check in lambda handler

    commit 77b972a1b2069e039c5067eb7dd2e95bcb2acea1
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 16:10:29 2022 +0100

        Switched to using ThreadPoolExecutor

    commit 4feb1b65d6ec0fcff2854b7080f8995e6ae77fe1
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 15:01:12 2022 +0100

        Optimized image downloads

    commit 2819b504a36f7b39d61640b87cb0a81e804e36a7
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 02:01:43 2022 +0100

        Add quality argument

    commit 2399788fdc0d8a10063f30c448462e9caa79997f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 01:29:47 2022 +0100

        Misc code cleanup; Fixed Discord not embedding 301 redirects issue

    commit 7a021916f0c9e8404646104258394e764080bffc
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Thu May 26 00:58:28 2022 +0100

        Use proper mime type

    commit a06a89825cf29d07787286d4837dfcb7a0151760
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:56:48 2022 +0100

        Fixed formatting

    commit 6717fd464aaa934724ea9c9f6022402013c7238b
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:55:41 2022 +0100

        Documented combination_method

    commit 022edb5122ddef8a8b3f84ffe84b1b2e4cf8130f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 23:48:55 2022 +0100

        Moved combineImg to own module; created AWS dockerfile; added combination_method config

    commit d7e0cb90898ac1aacfbe600f6dd1d8137185791a
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 17:28:03 2022 +0100

        Changed scaling method

    commit 55ea554357aca026926e1d296fb5ac259458c1d3
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 16:30:51 2022 +0100

        Re-scale image down

    commit b457760fae648c3cad5e383717f4df28c4539303
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 01:20:27 2022 +0100

        Use JPEG for final combined img

    commit 39a9ffd5120f7d28711e071290ee00d495a477ef
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 25 00:49:22 2022 +0100

        Method to embed combined tweet images. WIP & may be CPU intensive

    commit a6efd26447efa9b54e02c48bda9b58b038236ae0
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 23 20:33:11 2022 +0100

        Oops- Updated other two error msgs

    commit e81beb975ce319ccc95fa8db9e3e39927e205929
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 23 19:02:54 2022 +0100

        Improved Fail link message

    commit 55c956ad4ec7fe0ef0cbfb0e740bfcb1783aa88f
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 22 15:19:28 2022 +0100

        Error checking for loading cache json file

    commit 014f9d0e8d94da562f834665741adda744c0b42c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 22 15:19:17 2022 +0100

        TTL for cache entries

    commit f640ac69a1ba7f19ad69f6fea2199bec696d5eaf
    Merge: 7eff745 08d686c
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sat May 21 16:13:26 2022 +0100

        Merge pull request #2 from cabiste69/patch-1

        Minor cleaning

    commit 08d686c652a5edbf8252771c30a53743b5541f33
    Author: cabiste <ahmedjj354@gmail.com>
    Date:   Fri May 20 23:14:58 2022 +0100

        minor cleaning

    commit 7eff745550631e2d0ad014de9e941151bb5a8b61
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Wed May 18 18:23:11 2022 +0100

        Clarifications in readme

    commit 8df211090c4d73c812b76dba65c6dc180c848300
    Merge: 5a2a4e6 681f191
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Tue May 17 17:14:30 2022 +0100

        Merge pull request #1 from Fam0r/main

        Use the best bitrate video for twitter embeds

    commit 5a2a4e6316dfd1369b7a428ed4c9fcb9fe794de5
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Tue May 17 14:55:49 2022 +0100

        Display how many total images in the post

    commit 681f19150219c22a22999ac1e41dc5d393d5b9b5
    Author: Fam0r <fam0r@mailbox.org>
    Date:   Tue May 17 09:40:40 2022 +0300

        Use the best bitrate video for twitter embeds

    commit 82fe66af48c5dc4f8d933f2874f8bae815e7dd05
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Mon May 16 20:18:18 2022 +0100

        Work in progress code for combining up to 4 images

    commit fba1256efd3ce539f716e2f23b45c122c8422759
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:42:22 2022 +0100

        Fixed infinite redirect(?)

    commit 85c682ab111f575c8c517117f456cf354e9f5bf7
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:15:15 2022 +0100

        Updated gitignore

commit f2eefbe5a542f1a6fa8404228c2a9425054765a8
Author: Dylan <dylanpdx@gmail.com>
Date:   Wed May 18 18:21:00 2022 +0100

    Fixed DynamoDB integration

commit 32e5376b987c732147ab3289c2c42a8afde19e2f
Author: Dylan <dylanpdx@gmail.com>
Date:   Mon May 16 19:28:31 2022 +0100

    DynamoDB support & Serverless config update

commit dee62f812a6c8da443b111c7f3dd24bcb4d76525
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 17:06:42 2022 +0100

    Updated serverless.yml to dockerize pip packages

commit 43d69a1d872c293a4b70d2a330261245783eb924
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 14:30:18 2022 +0100

    serverless wsgi files

commit 96246aa9217774ddc88275ffba5bf1790317eda9
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 01:47:20 2022 +0100

    Squashed commit of the following:

    commit fba1256efd3ce539f716e2f23b45c122c8422759
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:42:22 2022 +0100

        Fixed infinite redirect(?)

    commit 85c682ab111f575c8c517117f456cf354e9f5bf7
    Author: Dylan <dylanpdx@gmail.com>
    Date:   Sun May 15 01:15:15 2022 +0100

        Updated gitignore

commit 2c9563fafe97bfe7782386aaa04372dcac5f8136
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 15 01:14:41 2022 +0100

    Serverless config
2022-09-12 17:36:11 +01:00
2022-09-01 22:33:07 +01:00
2022-09-12 17:36:11 +01:00
2021-07-03 18:52:30 -05:00
2022-09-12 17:36:11 +01:00
2022-09-01 23:54:15 +01:00
2022-09-01 21:38:42 +01:00
2022-05-13 17:00:16 +00:00
2022-09-12 17:36:11 +01:00
2022-06-01 19:25:23 +00:00

vxTwitter

(A fork of TwitFix) Basic flask server that serves fixed twitter video embeds to desktop discord by using either the Twitter API or Youtube-DL to grab tweet video information. This also automatically embeds the first link in the text of non video tweets (API Only)

Differences from fxtwitter

fxtwitter exposed all recently processed tweets publicly via a "latest" and "top" page.

Even though Tweets are public, it was a personal concern for me that a tweet with potentially sensitive information in it could suddenly be shown to however many people were browsing the latest tweets page, and could be used as a tool for harassment. This was removed in The following commit

How to use (discord side)

just put the url to the server, and directly after, the full URL to the tweet you want to embed

I now have a copy of this running on a Linode server, you can use it via the following url

https://vxtwitter.com/[twitter video url] or [last half of twitter url] (everything past twitter.com/)

You can also simply type out 'vx' directly before 'twitter.com' in any valid twitter video url, and that will convert it into a working vxTwitter url, for example:

Note: If you enjoy this service, please considering donating via Ko-Fi to help cover server costs

I do not monitor any tweets processed by this server. Additionally, if you plan on hosting the code yourself and are concerned about this, be sure to check how to disable logging on the web server you are using (i.e Nginx)

How to run (server side)

this script uses the youtube-dl python module, along with flask, twitter and pymongo, so install those with pip (you can use pip install -r requirements.txt) and start the server with python twitfix.py

I have included some files to give you a head start on setting this server up with uWSGI, though if you decide to use uWSGI I suggest you set up mongoDB link caching

Config

vxTwitter generates a config.json in its root directory the first time you run it, the options are:

API - This will be where you put the credentials for your twitter API if you use this method

database - This is where you put the URL to your mongoDB database if you are using one

link_cache - (Options: db, json)

  • db: Caches all links to a mongoDB database. This should be used it you are using uWSGI and are not just running the script on its own as one worker
  • json: This saves cached links to a local links.json file
  • none: Does not cache requests. Not reccomended as you can easily use up your Twitter API credits with this. Intended for use with another cache system (i.e NGINX uwsgi_cache)

method - ( Options: youtube-dl, api, hybrid )

  • youtube-dl: the original method for grabbing twitter video links, this uses a guest token provided via youtube-dl and should work well for individual instances, but may not scale up to a very large amount of usage

  • api: this directly uses the twitter API to grab tweet info, limited to 900 calls per 15m

  • hybrid: This will start off by using the twitter API to grab tweet info, but if the rate limit is reached or the api fails for any other reason it will switch over to youtube-dl to avoid downtime

color - Accepts a hex formatted color code, can change the embed color

appname - Can change the app name easily wherever it's shown

repo - used to change the repo url that some links redirect to

url - used to tell the user where to look for the oembed endpoint, make sure to set this to your public facing url

combination_method - using c.vxtwitter as the url causes vxTwitter to combine all images in the post into one. This is CPU intensive, so you might not want it running on the same machine that's serving requests. When combination_method is set to local, it will use the local machine to combine the images. This requires pillow to be installed. If you want to use another server, replace local with the URL to the endpoint which combines images. Both methods use the code in the combineImg module. Inside, there's also a Dockerfile intended to be deployed as a combination endpoint on an AWS Lambda function.

apiMirrors - During an influx of traffic (i.e when Twitter embeds break!) it's very likely that Twitter will begin to block you for sending too many requests. This is an array of replacement Twitter API URLs which can be called upon when requesting tweet info locally fails. An example of an implementation of an API Mirror is in the twExtract directory, and is intended to be hosted on AWS Lambda. If multiple mirror URLs are specified, one will be selected at random.

This project is licensed under the Do What The Fuck You Want Public License

Other stuff

We check for t.co links in non video tweets, and if one is found, we direct the discord useragent to embed that link directly, this means that twitter links containing youtube / vimeo links will automatically embed those as if you had just directly linked to that content

Description
Our fork of BetterTwitFix
Readme 14 MiB
Languages
Python 83.9%
HTML 9.6%
CSS 2.8%
Dockerfile 2%
Shell 1%
Other 0.7%