alterware/BetterTwitFix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit aae42b88c9
Merge: f2cec85 ab59881
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Jun 15 09:16:46 2023 +0100

    Merge branch 'main' of github.com:dylanpdx/BetterTwitFix

commit f2cec8521f
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu Jun 15 09:16:41 2023 +0100

    Very very quick fix for API changes

commit ab5988148c
Merge: 16f133a 1511551
Author: Dylan <dylanpdx@gmail.com>
Date:   Sat Jun 3 15:04:23 2023 +0100

    Merge pull request #98 from dylanpdx/dependabot/pip/boto3-1.26.146

    Bump boto3 from 1.26.133 to 1.26.146

commit 1511551719
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat Jun 3 13:34:51 2023 +0000

    Bump boto3 from 1.26.133 to 1.26.146

    Bumps [boto3](https://github.com/boto/boto3) from 1.26.133 to 1.26.146.
    - [Release notes](https://github.com/boto/boto3/releases)
    - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
    - [Commits](https://github.com/boto/boto3/compare/1.26.133...1.26.146)

    ---
    updated-dependencies:
    - dependency-name: boto3
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 16f133a11b
Merge: 9f8f3ea 6b884d4
Author: Dylan <dylanpdx@gmail.com>
Date:   Sat Jun 3 14:34:10 2023 +0100

    Merge pull request #96 from dylanpdx/dependabot/pip/requests-2.31.0

    Bump requests from 2.30.0 to 2.31.0

commit 6b884d42b0
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon May 29 02:00:29 2023 +0000

    Bump requests from 2.30.0 to 2.31.0

    Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0.
    - [Release notes](https://github.com/psf/requests/releases)
    - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
    - [Commits](https://github.com/psf/requests/compare/v2.30.0...v2.31.0)

    ---
    updated-dependencies:
    - dependency-name: requests
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 9f8f3ea41e
Merge: cda291f a710867
Author: Dylan <dylanpdx@gmail.com>
Date:   Tue May 16 15:44:07 2023 +0100

    Merge branch 'prod' into main

commit cda291fe9e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue May 16 15:42:18 2023 +0100

    Bump boto3 from 1.26.129 to 1.26.133 (#92)

    Bumps [boto3](https://github.com/boto/boto3) from 1.26.129 to 1.26.133.
    - [Release notes](https://github.com/boto/boto3/releases)
    - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
    - [Commits](https://github.com/boto/boto3/compare/1.26.129...1.26.133)

    ---
    updated-dependencies:
    - dependency-name: boto3
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 5cb8d479f5
Author: Dylan <dylanpdx@gmail.com>
Date:   Tue May 16 15:40:34 2023 +0100

    Fixed QRTs sometimes failing to embed

commit b7f285c2cd
Merge: 66c0044 49ec005
Author: Dylan <dylanpdx@gmail.com>
Date:   Tue May 16 15:32:28 2023 +0100

    Merge branch 'main' of github.com:dylanpdx/BetterTwitFix

commit 66c004453b
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun May 14 22:04:35 2023 +0100

    Increased timeout to 15

commit 49ec0059cd
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue May 9 16:53:02 2023 +0100

    Bump boto3 from 1.26.127 to 1.26.129 (#90)

    Bumps [boto3](https://github.com/boto/boto3) from 1.26.127 to 1.26.129.
    - [Release notes](https://github.com/boto/boto3/releases)
    - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
    - [Commits](https://github.com/boto/boto3/compare/1.26.127...1.26.129)

    ---
    updated-dependencies:
    - dependency-name: boto3
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c395c00a11
Author: Dylan <dylanpdx@gmail.com>
Date:   Fri May 5 14:08:23 2023 +0100

    Fix text trimming with images and text posts

commit c6840ad093
Author: Dylan <dylanpdx@gmail.com>
Date:   Fri May 5 13:56:45 2023 +0100

    Revert "Fix text trimming w/ old limits"

    This reverts commit 9d9c32b797.

commit 9d9c32b797
Author: Dylan <dylanpdx@gmail.com>
Date:   Fri May 5 13:46:42 2023 +0100

    Fix text trimming w/ old limits

commit dda1187f4a
Merge: 14c9426 f0065ce
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:48:42 2023 +0100

    Merge branch 'prod' into main

commit 14c942645c
Merge: d621bfc e895e4c
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:45:19 2023 +0100

    Merge pull request #84 from dylanpdx/dependabot/pip/requests-2.30.0

    Bump requests from 2.28.2 to 2.30.0

commit e895e4c876
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu May 4 22:44:08 2023 +0000

    Bump requests from 2.28.2 to 2.30.0

    Bumps [requests](https://github.com/psf/requests) from 2.28.2 to 2.30.0.
    - [Release notes](https://github.com/psf/requests/releases)
    - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
    - [Commits](https://github.com/psf/requests/compare/v2.28.2...v2.30.0)

    ---
    updated-dependencies:
    - dependency-name: requests
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit d621bfc8a8
Merge: 727d144 98e006b
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:43:28 2023 +0100

    Merge pull request #86 from dylanpdx/dependabot/pip/boto3-1.26.127

    Bump boto3 from 1.26.104 to 1.26.127

commit 727d144ece
Merge: 1a698c1 618a644
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:42:04 2023 +0100

    Merge pull request #87 from dylanpdx/dependabot/npm_and_yarn/simple-git-3.18.0

    Bump simple-git from 3.7.1 to 3.18.0

commit 618a644070
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu May 4 22:40:58 2023 +0000

    Bump simple-git from 3.7.1 to 3.18.0

    Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.7.1 to 3.18.0.
    - [Release notes](https://github.com/steveukx/git-js/releases)
    - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md)
    - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.18.0/simple-git)

    ---
    updated-dependencies:
    - dependency-name: simple-git
      dependency-type: indirect
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 98e006b31c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu May 4 22:39:39 2023 +0000

    Bump boto3 from 1.26.104 to 1.26.127

    Bumps [boto3](https://github.com/boto/boto3) from 1.26.104 to 1.26.127.
    - [Release notes](https://github.com/boto/boto3/releases)
    - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
    - [Commits](https://github.com/boto/boto3/compare/1.26.104...1.26.127)

    ---
    updated-dependencies:
    - dependency-name: boto3
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 1a698c1d6f
Merge: a6eb39e 8205ef2
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:38:58 2023 +0100

    Merge pull request #71 from dylanpdx/dependabot/pip/pillow-9.5.0

    Bump pillow from 9.4.0 to 9.5.0

commit a6eb39e545
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:35:52 2023 +0100

    Fix tweet not found test

commit d84938ee08
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:29:12 2023 +0100

    Fixed tweet not found error

commit 69b99dcc9b
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:08:11 2023 +0100

    Revert "Add info message for #74"

    This reverts commit 39490a49b9.

commit 8674c8d828
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 23:07:15 2023 +0100

    Fix/workarounds for failing tests

commit 3127fff1ae
Author: Dylan <dylanpdx@gmail.com>
Date:   Thu May 4 22:45:55 2023 +0100

    Update gitignore

commit 6244c875c4
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun Apr 9 23:11:31 2023 +0100

    Fix small embed sizes

commit edfeefbc00
Author: Dylan <dylanpdx@gmail.com>
Date:   Sun Apr 9 22:26:38 2023 +0100

    Fix NSFW video embeds

commit dd961df082
Author: Dylan <dylanpdx@gmail.com>
Date:   Sat Apr 8 18:11:14 2023 +0100

    Update deploy workflow

commit 8205ef2400
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat Apr 8 16:59:27 2023 +0000

    Bump pillow from 9.4.0 to 9.5.0

    Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.4.0 to 9.5.0.
    - [Release notes](https://github.com/python-pillow/Pillow/releases)
    - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
    - [Commits](https://github.com/python-pillow/Pillow/compare/9.4.0...9.5.0)

    ---
    updated-dependencies:
    - dependency-name: pillow
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
This commit is contained in:
Dylan
2023-06-15 09:17:39 +01:00
parent 1986b52ea6
commit 6b13c1cd0f
5 changed files with 27 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.github/workflows/deploy.yml vendored
View File

@@ -69,3 +69,4 @@ jobs:
VXTWITTER_WORKAROUND_CONSUMER_SECRET: ${{ secrets.VXTWITTER_WORKAROUND_CONSUMER_SECRET }} VXTWITTER_WORKAROUND_CONSUMER_SECRET: ${{ secrets.VXTWITTER_WORKAROUND_CONSUMER_SECRET }}
VXTWITTER_WORKAROUND_TOKEN: ${{ secrets.VXTWITTER_WORKAROUND_TOKEN }} VXTWITTER_WORKAROUND_TOKEN: ${{ secrets.VXTWITTER_WORKAROUND_TOKEN }}
VXTWITTER_WORKAROUND_TOKEN_SECRET: ${{ secrets.VXTWITTER_WORKAROUND_TOKEN_SECRET }} VXTWITTER_WORKAROUND_TOKEN_SECRET: ${{ secrets.VXTWITTER_WORKAROUND_TOKEN_SECRET }}
VXTWITTER_WORKAROUND_TOKENS: ${{ secrets.VXTWITTER_WORKAROUND_TOKENS }}

1
configHandler.py
View File

@@ -15,6 +15,7 @@ elif ('RUNNING_SERVERLESS' in os.environ and os.environ['RUNNING_SERVERLESS'] ==
"url": os.environ["VXTWITTER_URL"], "url": os.environ["VXTWITTER_URL"],
"combination_method": os.environ["VXTWITTER_COMBINATION_METHOD"], # can either be 'local' or a URL to a server handling requests in the same format "combination_method": os.environ["VXTWITTER_COMBINATION_METHOD"], # can either be 'local' or a URL to a server handling requests in the same format
"gifConvertAPI":os.environ["VXTWITTER_GIF_CONVERT_API"], "gifConvertAPI":os.environ["VXTWITTER_GIF_CONVERT_API"],
"workaroundTokens":os.environ["VXTWITTER_WORKAROUND_TOKENS"],
"workaroundKeys":{ "workaroundKeys":{
"consumerKey":os.environ["VXTWITTER_WORKAROUND_CONSUMER_KEY"], "consumerKey":os.environ["VXTWITTER_WORKAROUND_CONSUMER_KEY"],
"consumerSecret":os.environ["VXTWITTER_WORKAROUND_CONSUMER_SECRET"], "consumerSecret":os.environ["VXTWITTER_WORKAROUND_CONSUMER_SECRET"],

4
requirements.txt
View File

@@ -1,6 +1,6 @@
pymongo==4.3.3 pymongo==4.3.3
boto3==1.26.133 boto3==1.26.146
requests==2.30.0 requests==2.31.0
Pillow==9.5.0 Pillow==9.5.0
Flask==2.2.3 Flask==2.2.3
Flask-Cors==3.0.10 Flask-Cors==3.0.10

1
serverless.yml
View File

@@ -31,6 +31,7 @@ provider:
VXTWITTER_WORKAROUND_CONSUMER_SECRET: ${env:VXTWITTER_WORKAROUND_CONSUMER_SECRET, ''} VXTWITTER_WORKAROUND_CONSUMER_SECRET: ${env:VXTWITTER_WORKAROUND_CONSUMER_SECRET, ''}
VXTWITTER_WORKAROUND_TOKEN: ${env:VXTWITTER_WORKAROUND_TOKEN, ''} VXTWITTER_WORKAROUND_TOKEN: ${env:VXTWITTER_WORKAROUND_TOKEN, ''}
VXTWITTER_WORKAROUND_TOKEN_SECRET: ${env:VXTWITTER_WORKAROUND_TOKEN_SECRET, ''} VXTWITTER_WORKAROUND_TOKEN_SECRET: ${env:VXTWITTER_WORKAROUND_TOKEN_SECRET, ''}
VXTWITTER_WORKAROUND_TOKENS: ${env:VXTWITTER_WORKAROUND_TOKENS, ''}
package: package:
patterns: patterns:

43
twExtract/__init__.py
View File

@@ -1,26 +1,17 @@
import yt_dlp import yt_dlp
from yt_dlp.extractor import twitter from yt_dlp.extractor import twitter
import uuid
import json import json
import requests import requests
import re import re
import random
from . import twExtractError from . import twExtractError
import twitter
from configHandler import config from configHandler import config
bearer="Bearer AAAAAAAAAAAAAAAAAAAAAPYXBAAAAAAACLXUNDekMxqa8h%2F40K4moUkGsoc%3DTYfbDKbT3jJPCEVnMYqilB28NHfOPqkca3qaAxGfsyKCs0wRbw" bearer="Bearer AAAAAAAAAAAAAAAAAAAAAPYXBAAAAAAACLXUNDekMxqa8h%2F40K4moUkGsoc%3DTYfbDKbT3jJPCEVnMYqilB28NHfOPqkca3qaAxGfsyKCs0wRbw"
guestToken=None guestToken=None
pathregex = r"\w{1,15}\/(status|statuses)\/(\d{2,20})" pathregex = r"\w{1,15}\/(status|statuses)\/(\d{2,20})"
userregex = r"^https?:\/\/(?:www\.)?twitter\.com\/(?:#!\/)?@?([^/?#]*)(?:[?#/].*)?$" userregex = r"^https?:\/\/(?:www\.)?twitter\.com\/(?:#!\/)?@?([^/?#]*)(?:[?#/].*)?$"
userIDregex = r"\/i\/user\/(\d+)" userIDregex = r"\/i\/user\/(\d+)"
try:
auth = twitter.oauth.OAuth(
config['config']['workaroundKeys']["accessToken"],
config['config']['workaroundKeys']["accessTokenSecret"],
config['config']['workaroundKeys']["consumerKey"],
config['config']['workaroundKeys']["consumerSecret"]
)
api = twitter.Twitter(auth=auth)
except Exception as e:
api = None
def getGuestToken(): def getGuestToken():
global guestToken global guestToken
@@ -30,17 +21,25 @@ def getGuestToken():
return guestToken return guestToken
def extractStatus_fallback(url): def extractStatus_fallback(url):
if api is None: try:
raise twExtractError.TwExtractError(500, "Could not extract tweet.") # get tweet ID
print(" ➤ [ I ] Using fallback method to extract tweet") m = re.search(pathregex, url)
# get tweet ID if m is None:
m = re.search(pathregex, url) raise twExtractError.TwExtractError(400, "Extract error")
if m is None: twid = m.group(2)
raise twExtractError.TwExtractError(400, "Invalid URL") # get tweet
twid = m.group(2)
# get tweet authToken=random.choice(config["config"]["workaroundTokens"].split(","))
tweet = api.statuses.show(_id=twid, tweet_mode="extended") csrfToken=str(uuid.uuid4()).replace('-', '')
return tweet tweet = requests.get("https://api.twitter.com/1.1/statuses/show/" + twid + ".json?tweet_mode=extended&cards_platform=Web-12&include_cards=1&include_reply_count=1&include_user_entities=0", headers={"Authorization":bearer,"Cookie":f"auth_token={authToken}; ct0={csrfToken}; ","x-twitter-active-user":"yes","x-twitter-auth-type":"OAuth2Session","x-twitter-client-language":"en","x-csrf-token":csrfToken,"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0"})
output = tweet.json()
if "errors" in output:
# pick the first error and create a twExtractError
error = output["errors"][0]
raise twExtractError.TwExtractError(error["code"], error["message"])
return output
except Exception as e:
raise twExtractError.TwExtractError(400, "Extract error")
def extractStatus(url): def extractStatus(url):