contentDetails -> duration always returns 0
#299
Assignees
Labels
bug
Something isn't working
medium priority
A high-priority issue noticeable by the user but he can still work around it.
medium
A task that should take less than a day to complete.
official instance
Comments
Benjamin-Loison
added
bug
Benjamin-Loison
changed the title
contentDetails -> duration always returns 0
|
The issue only happens on the official instance, I think it is related to #11, hosting your own instance is a workaround, I will investigate such that the official instance at least returns an error. |
|
Humm, It used to work very well before. See this post from may 28 2024 - cn-tools/cntools_FreshRssExtensions#8 (comment) #11 is 2 years old. |
|
I know that it worked before. #11 is just the issue where I gather information about YouTube detecting the instance as automated, hence blocking it, which seems to be the case here as local instances work fine. |
|
Right, thanks for the precision. Appreciated. |
|
Viewing in Firefox
I will check if #296 patch still works fine in its context and here. #296 patch seems to work fine in its context: VIDEO_IDS=('UPrkC1LdlLY' 'a9cyG_yfh1k')
YOUTUBE_OPERATIONAL_API_INSTANCE_URL='https://yt.lemnoslife.com'
for videoId in "${VIDEO_IDS[@]}"
do
echo $videoId
curl -s "$YOUTUBE_OPERATIONAL_API_INSTANCE_URL/videos?part=music&id=$videoId" | jq '.items[0].music.available' -
doneSource: #296#issuecomment-2278630907 getJSONPathFromKey.py n8vmXvoVjZw '' ytInitialPlayerResponse | grep -i robotSame string so let us use curl -H 'Accept-Language: en' 'https://www.youtube.com/watch?v=n8vmXvoVjZw' > n8vmXvoVjZwSource: blob/c4539bc55d74c556570e1eabb710369a7fca159c/common.php#L178 getJSONPathFromKey.py n8vmXvoVjZw reason ytInitialPlayerResponseUsing the following I do not know which one was used in #296. minimizeCURL.py curl.sh '"lengthSeconds":"1222"'#24 would precise exhaustively the requests requiring authentication. Can otherwise send myself a message once someone makes a request on the official instance triggering this issue. As shown in Benjamin-Loison/matrix-commander/issues/16 I may be spammed but not too much and Benjamin-Loison/matrix-commander/issues/8 can help anyway. grep -r 'file_get_contents(' | grep -v "file_get_contents('tests/"grep -r 'fileGetContentsAndHeadersFromOpts('curl -I 'https://www.youtube.com/watch?v=n8vmXvoVjZw'returns grep -rn 'getRemote('Following e179590: grep -rn 'getRemote('It seems that we can focus on videos and more especially on endpoints requiring the video to load. So the following does not seem necessary, should focus on endpoints potentially suffering of the issue, so YouTube operational API diff --git a/common.php b/common.php
index 50fe0c2..c58f2ef 100644
--- a/common.php
+++ b/common.php
@@ -168,6 +168,8 @@
return getJSONFromHTML($url, $opts, $scriptVariable, $prefix, $forceLanguage, $verifiesChannelRedirection);
}
}
+ // #24 would maybe reduce the following need:
+ //if(in_array($scriptVariable, ['', 'ytInitialData']))
return $json;
}so can leverage and complete unit tests of: php test.php videos 2> /dev/nullIt is not a problem that some unit test fail both locally and on the official instance as it means that the associated feature does not seem to work anymore. On my laptop: php test.php videos 2> /dev/nullSo there is a single On the official instance: php test.php videos 2> /dev/nullSo there are 5 other minimizeCURL.py curl.sh paidContentOverlayBeing precise to reduce requests leveraging authentication may help not make these credentials being detected as automated. Concerning {
"responseContext": {
"visitorData": "CgtEQ0tkRnNLal9zQSic6tO2BjIiCgJGUhIcEhgSFgsMDg8QERITFBUWFxgZGhscHR4fICEgSg%3D%3D",
"serviceTrackingParams": [
{
"service": "GFEEDBACK",
"params": [
{
"key": "is_alc_surface",
"value": "false"
},
{
"key": "is_viewed_live",
"value": "False"
},
{
"key": "ipcc",
"value": "0"
},
{
"key": "logged_in",
"value": "0"
},
{
"key": "e",
"value": "23804281,23966208,23998056,24004644,24077241,24181174,24241378,24290971,24425063,24439361,24468724,24542367,24548629,24566687,51009781,51010235,51017346,51020570,51025415,51030103,51037342,51037353,51041512,51050361,51053689,51057844,51057851,51063643,51064835,51098297,51098299,51111738,51115184,51116067,51123611,51124104,51133103,51138234,51149607,51152050,51157411,51157841,51158514,51160545,51162170,51165467,51169118,51176511,51177012,51177817,51178314,51178327,51178346,51178355,51178982,51183910,51184022,51186528,51190057,51190073,51190082,51190085,51190200,51190209,51190220,51190229,51190652,51195231,51196181,51196478,51197687,51197694,51197701,51197706,51200253,51200260,51200293,51200300,51201350,51201365,51201372,51201381,51201426,51201433,51201440,51201447,51204329,51209050,51212458,51212466,51212546,51212555,51212569,51217504,51219800,51221011,51221150,51222972,51223962,51224135,51227037,51227291,51227410,51227776,51228350,51228352,51228767,51228778,51228783,51228796,51228809,51228812,51230241,51230389,51230478,51231373,51231814,51234407,51235080,51236017,51237842,51239093,51241028,51242269,51242447,51243940,51245822,51245831,51246283,51246305,51248255,51248734,51248739,51248747,51248748,51249769,51251508,51251675,51251811,51251836,51251849,51255677,51255681,51255743,51256074,51256084,51256732,51257533,51257852,51257902,51257918,51258066,51258360,51258457,51258835,51259133,51260592,51260634,51264983,51265341,51265356,51265377,51266743,51266946,51267567,51268387,51268978,51269632,51270086,51270830,51271256,51271390,51271669,51272491,51272506,51272513,51272530,51272570,51272589,51272663,51272710,51273423,51273446,51275152,51275157,51275172,51275185,51275194,51275206,51277508,51278034,51280249,51281600,51282073,51282084,51282508,51284157,51285417"
}
]
},
{
"service": "CSI",
"params": [
{
"key": "c",
"value": "WEB_EMBEDDED_PLAYER"
},
{
"key": "cver",
"value": "1.9999099"
},
{
"key": "yt_li",
"value": "0"
},
{
"key": "GetPlayer_rid",
"value": "0xa826c5e2625c823f"
}
]
},
{
"service": "GUIDED_HELP",
"params": [
{
"key": "logged_in",
"value": "0"
}
]
},
{
"service": "ECATCHER",
"params": [
{
"key": "client.version",
"value": "20240902"
},
{
"key": "client.name",
"value": "WEB_EMBEDDED_PLAYER"
}
]
}
],
"maxAgeSeconds": 0
},
"playabilityStatus": {
"status": "LOGIN_REQUIRED",
"reason": "Sign in to confirm you’re not a bot",
"errorScreen": {
"playerErrorMessageRenderer": {
"subreason": {
"runs": [
{
"text": "This helps protect our community. "
},
{
"text": "Learn more",
"navigationEndpoint": {
"clickTrackingParams": "CAAQu2kiEwjGjenI76KIAxXFZE8EHadtFNc=",
"urlEndpoint": {
"url": "https://support.google.com/youtube/answer/3037019#zippy=%2Ccheck-that-youre-signed-into-youtube"
}
}
}
]
},
"reason": {
"runs": [
{
"text": "Sign in to confirm you’re not a bot"
}
]
},
"proceedButton": {
"buttonRenderer": {
"trackingParams": "CAEQ8FsiEwjGjenI76KIAxXFZE8EHadtFNc="
}
},
"thumbnail": {
"thumbnails": [
{
"url": "//s.ytimg.com/yts/img/meh7-vflGevej7.png",
"width": 140,
"height": 100
}
]
},
"icon": {
"iconType": "ERROR_OUTLINE"
}
}
},
"contextParams": "Q0FFU0FnZ0M="
},
"trackingParams": "CAAQu2kiEwjGjenI76KIAxXFZE8EHadtFNc=",
"adBreakHeartbeatParams": "Q0FBJTNE"
}With: diff --git a/common.php b/common.php
index 50fe0c2..c8a2d1e 100644
--- a/common.php
+++ b/common.php
@@ -19,10 +19,24 @@
}
}
- function getContextFromOpts($opts)
+ function implodeArray($anArray, $separator)
{
- if (GOOGLE_ABUSE_EXEMPTION !== '') {
- $cookieToAdd = 'GOOGLE_ABUSE_EXEMPTION=' . GOOGLE_ABUSE_EXEMPTION;
+ return array_map(fn($k, $v) => "${k}${separator}${v}", array_keys($anArray), array_values($anArray));
+ }
+
+ function getContextFromOpts($opts, $mayRequireAuthentication = false)
+ {
+ if (GOOGLE_ABUSE_EXEMPTION !== '' or $mayRequireAuthentication) {
+ $cookiesToAdd = [];
+ if (GOOGLE_ABUSE_EXEMPTION !== '') {
+ $cookiesToAdd['GOOGLE_ABUSE_EXEMPTION'] = GOOGLE_ABUSE_EXEMPTION;
+ }
+ if ($mayRequireAuthentication) {
+ // Both or equivalent `1` instead of `3` are needed for `videos?part=isPaidPromotion&id=PEorJqo2Qaw`.
+ $cookiesToAdd['__Secure-3PSID'] = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
+ $cookiesToAdd['__Secure-3PSIDTS'] = 'sidts-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
+ }
+ $cookiesToAdd = implode('; ', implodeArray($cookiesToAdd, '='));
// Can't we simplify the following code?
if (array_key_exists('http', $opts)) {
$http = $opts['http'];
@@ -31,20 +45,20 @@
$isThereACookieHeader = false;
foreach ($headers as $headerIndex => $header) {
if (str_starts_with($header, 'Cookie: ')) {
- $opts['http']['header'][$headerIndex] = "$header; $cookieToAdd";
+ $opts['http']['header'][$headerIndex] = "$header; $cookiesToAdd";
$isThereACookieHeader = true;
break;
}
}
if (!$isThereACookieHeader) {
- array_push($opts['http']['header'], "Cookie: $cookieToAdd");
+ array_push($opts['http']['header'], "Cookie: $cookiesToAdd");
}
}
} else {
$opts = [
'http' => [
'header' => [
- "Cookie: $cookieToAdd"
+ "Cookie: $cookiesToAdd"
]
]
];
@@ -61,7 +75,7 @@
return $headers;
}
- function fileGetContentsAndHeadersFromOpts($url, $opts)
+ function fileGetContentsAndHeadersFromOpts($url, $opts, $mayRequireAuthentication)
{
if(HTTPS_PROXY_ADDRESS !== '')
{
@@ -78,7 +92,7 @@
$opts['http']['header'] = $headers;
}
}
- $context = getContextFromOpts($opts);
+ $context = getContextFromOpts($opts, $mayRequireAuthentication);
$result = file_get_contents($url, false, $context);
return [$result, $http_response_header];
}
@@ -98,9 +112,9 @@
return $code == 303;
}
- function getRemote($url, $opts = [], $verifyTrafficIfForbidden = true)
+ function getRemote($url, $opts = [], $verifyTrafficIfForbidden = true, $mayRequireAuthentication = false)
{
- [$result, $headers] = fileGetContentsAndHeadersFromOpts($url, $opts);
+ [$result, $headers] = fileGetContentsAndHeadersFromOpts($url, $opts, $mayRequireAuthentication);
foreach (HTTP_CODES_DETECTED_AS_SENDING_UNUSUAL_TRAFFIC as $HTTP_CODE_DETECTED_AS_SENDING_UNUSUAL_TRAFFIC) {
if (str_contains($headers[0], strval($HTTP_CODE_DETECTED_AS_SENDING_UNUSUAL_TRAFFIC)) && ($HTTP_CODE_DETECTED_AS_SENDING_UNUSUAL_TRAFFIC != 403 || $verifyTrafficIfForbidden)) {
detectedAsSendingUnusualTraffic();
@@ -152,9 +166,9 @@
return getJSONStringFromHTMLScriptPrefix($html, "$prefix$scriptVariable = ");
}
- function getJSONFromHTML($url, $opts = [], $scriptVariable = '', $prefix = 'var ', $forceLanguage = false, $verifiesChannelRedirection = false)
+ function getJSONFromHTML($url, $opts = [], $scriptVariable = '', $prefix = 'var ', $forceLanguage = false, $verifiesChannelRedirection = false, $mayRequireAuthentication = false)
{
- $html = getRemote($url, $opts);
+ $html = getRemote($url, $opts, mayRequireAuthentication: $mayRequireAuthentication);
$jsonStr = getJSONStringFromHTML($html, $scriptVariable, $prefix);
$json = json_decode($jsonStr, true);
if($verifiesChannelRedirection)
diff --git a/videos.php b/videos.php
index b101d3c..41e7b78 100644
--- a/videos.php
+++ b/videos.php
@@ -98,9 +98,20 @@
'Content-Type: application/json',
'Accept-Language: en'
];
- if ($music) {
- array_push($headers, 'Referer: https://music.youtube.com');
- }
+ // As I am unable to reproduce a request to the specified URL for `!$music`, as the `$music` case, works let us use it for both ca
ses.
+ // Maybe can share `__Secure-3PSID` with `getContextFromOpts`.
+ $currentTime = time();
+ $SAPISID = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
+ $__Secure_3PSID = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
+ $ORIGIN = 'https://music.youtube.com';
+ $SAPISIDHASH = "${currentTime}_" . sha1("$currentTime $SAPISID $ORIGIN");
+
+ array_push($headers,
+ //'Referer: https://music.youtube.com',
+ "Origin: $ORIGIN",
+ "Authorization: SAPISIDHASH $SAPISIDHASH",
+ "Cookie: __Secure-3PSID=$__Secure_3PSID; __Secure-3PAPISID=$SAPISID",
+ );
$opts = [
'http' => [
'method' => 'POST',
@@ -266,7 +276,7 @@
}
if ($options['isPaidPromotion']) {
- $json = getJSONFromHTML("https://www.youtube.com/watch?v=$id", scriptVariable: 'ytInitialPlayerResponse');
+ $json = getJSONFromHTML("https://www.youtube.com/watch?v=$id", scriptVariable: 'ytInitialPlayerResponse', mayRequireAuthentication: true);
$isPaidPromotion = array_key_exists('paidContentOverlay', $json);
$item['isPaidPromotion'] = $isPaidPromotion;
}
@@ -326,7 +336,7 @@
}
if ($options['qualities']) {
- $json = getJSONFromHTML("https://www.youtube.com/watch?v=$id", scriptVariable: 'ytInitialPlayerResponse');
+ $json = getJSONFromHTML("https://www.youtube.com/watch?v=$id", scriptVariable: 'ytInitialPlayerResponse', mayRequireAuthentication: true);
$qualities = [];
foreach ($json['streamingData']['adaptiveFormats'] as $quality) {
if (array_key_exists('qualityLabel', $quality)) {
@@ -386,7 +396,7 @@
'header' => ['Cookie: PREF=f2=8000000'],
]
];
- $json = getJSONFromHTML("https://www.youtube.com/watch?v=$id", $opts, 'ytInitialPlayerResponse');
+ $json = getJSONFromHTML("https://www.youtube.com/watch?v=$id", $opts, 'ytInitialPlayerResponse', mayRequireAuthentication: true);
$playabilityStatus = $json['playabilityStatus'];
$isRestricted = array_key_exists('isBlockedInRestrictedMode', $playabilityStatus);
$item['isRestricted'] = $isRestricted;only get above local Being notified when the official instance answers an unexpected value could help but these would need to be exhaustive which is not the case of https://yt0.lemnoslife.com/metrics/ so let us postpone this, as current |
|
Interesting. I can confirm running a local instance works. |
Benjamin-Loison
added a commit
that referenced
this issue
Sep 1, 2024
|
Tried from home, just for test... |
|
There is no documentation for work in progress unit tests, so I was not expecting you to run them. If you remove Also note that some tests need actual JSON to match against, those JSON have not all yet been publicly published as I am looking for better ways to store these expectated JSONs. |
|
OK no problem. I'm just curious. Got more from apache error log. For each request: Was probably expecting Will add |
|
I guess you meant |
|
No, |
|
You can try |
|
Sure but not before a complete backup. Had terrible issues in the past on Debian. The 2 were conflicting. |
|
If you are curious, then know that current About the issue itself I logged in with an ad-hoc account on the official instance (see #issuecomment-2323470675) solving your issue: curl 'https://yt.lemnoslife.com/videos?part=contentDetails&id=n8vmXvoVjZw'{
"kind": "youtube#videoListResponse",
"etag": "NotImplemented",
"items": [
{
"kind": "youtube#video",
"etag": "NotImplemented",
"id": "n8vmXvoVjZw",
"contentDetails": {
"duration": 1222
}
}
]
}thank you for having reported this issue. |
Benjamin-Loison
added
medium
|
Thanks a lot! You're very welcome. Fix confirmed. |
|
Note that the issue may come back in the future, do not hesitate to re-open this issue if it is the case. #issuecomment-2323470675 mentions the missing proper monitoring. |
|
Related to YoutubeExplode/issues/794. |
|
Hi, It's now perfectly detected. Glad I installed it locally ;) {
"error": {
"code": 403,
"message": "YouTube has detected unusual traffic from this YouTube operational API instance. Please try your request again later or see alternatives at https:\/\/github.com\/Benjamin-Loison\/YouTube-operational-API\/issues\/11"
}
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For a couple of weeks, impossible to get the duration of a YT video. It always returns 0. Perfectly repeatable.
e.g. https://yt.lemnoslife.com/videos?part=contentDetails&id=n8vmXvoVjZw
{ "kind": "youtube#videoListResponse", "etag": "NotImplemented", "items": [ { "kind": "youtube#video", "etag": "NotImplemented", "id": "n8vmXvoVjZw", "contentDetails": { "duration": 0 } } ] }The text was updated successfully, but these errors were encountered: