JEF - All Critical

Test Results

Running test...

#66 Failed 429

2025-12-06 11:10:45 (6.29s)

AI: The test failed to retrieve the expected product data due to an API rate limiting error, resulting in a 429 status code instead of the anticipated 200 OK.

Expected status 200, got 429

AI Functional Analysis

Summary

The test failed to retrieve the expected product data due to an API rate limiting error, resulting in a 429 status code instead of the anticipated 200 OK.

Detailed Report

The test case 'JEF - All Critical - Test' resulted in a failure because the API call was throttled by a rate limiting policy, specifically a 'Spike arrest violation'. The actual response clearly indicates that the allowed rate of messages per period (30 messages per second with a burst of 3) was exceeded, leading to a 429 'Too Many Requests' status code. This prevented the application from returning the expected product details, documents, and compliance analysis. From a functional testing standpoint, this result is not acceptable as the application failed to deliver the expected business outcome (providing product information). However, the rate limiting mechanism itself appears to be functioning as designed to protect the API. The failure highlights an operational or infrastructure constraint rather than a bug in the application's internal data processing or retrieval logic. The key discrepancy is the complete absence of the expected data response. Instead of the detailed JSON object containing 'internal_id', 'mpn', 'documents', and 'compliance_analysis', the system returned an error object related to API usage. This indicates that the request never reached the point of functional processing, or was rejected at the gateway before it could be processed, thus preventing any functional validation of the data itself. To resolve this, the test execution strategy may need adjustment, such as introducing delays between requests or implementing retry logic, or the API rate limits in the test environment might need review.

{
  "fault": {
    "detail": {
      "errorcode": "policies.ratelimit.SpikeArrestViolation"
    },
    "faultstring": "Spike arrest violation. Allowed rate : MessageRate{messagesPerPeriod=30, periodInMicroseconds=1000000, maxBurstMessageCount=3.0}"
  }
}

{
  "compliance_analysis": null,
  "documents": [
    {
      "doc_metadata": {
        "compliance_status": "noncompliant",
        "coverage": "general",
        "date_of_issue": "2024-05-23",
        "is_approved": true,
        "revised_date": "2024-05-01",
        "version": "RoHS III"
      },
      "level": "Component",
      "type": "RoHS",
      "uploaded_at": "2025-12-30T11:49:58.074001",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Company",
      "type": "RoHS Exempt",
      "uploaded_at": "2025-12-05T11:49:58.074016",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf"
    },
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2025-09-02",
        "is_approved": true,
        "revised_date": "2025-08-26",
        "version": "REACH"
      },
      "level": "Company",
      "type": "REACH",
      "uploaded_at": "2025-12-05T11:49:58.074023",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Series",
      "type": "Datasheet",
      "uploaded_at": "2025-12-05T11:49:58.308975",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf"
    }
  ],
  "internal_id": "TI LM 99-358 - QX",
  "manufacturer": "Texas Instruments",
  "mpn": "LM358",
  "processing_time_ms": 0,
  "product_url": "https://www.ti.com/product/TILM99-358-QX",
  "status": "active"
}

#46 Failed 429

2025-12-06 11:10:44 (8.32s)

AI: The test case failed because the API returned a 'Rate limit exceeded' error (HTTP 429) instead of the expected product data (HTTP 200).

Expected status 200, got 429

AI Functional Analysis

Summary

The test case failed because the API returned a 'Rate limit exceeded' error (HTTP 429) instead of the expected product data (HTTP 200).

Detailed Report

The test case 'JEF - All Critical - Test' resulted in a failure. The system under test returned an HTTP 429 'Too Many Requests' status code with an 'Rate limit exceeded' error message, preventing the retrieval of the expected product information. The actual response indicated that the request limit of 10 requests within 60 seconds was breached, with a recommended retry after 60 seconds. This error completely prevented the functional validation of the API's ability to provide critical product data and compliance information as defined by the expected response. From a functional perspective, this result is unacceptable. The primary goal of this test is to verify the correct data retrieval and structure. The rate limit error obstructs this verification entirely. While rate limiting is a valid mechanism for API protection, its occurrence during a functional test run suggests an issue with the test environment, the test runner's request patterns, or the system's ability to scale for testing. Discrepancies include the HTTP status code (Expected 200, Actual 429) and the response body (Expected detailed product JSON, Actual error JSON). The failure points to an operational or infrastructure issue rather than an incorrect data processing issue. To resolve this, it's crucial to investigate why the rate limit was hit, potentially adjusting test runner delays, increasing rate limits in the test environment, or ensuring dedicated resources for test execution.

{
  "details": {
    "limit": 10,
    "retry_after": 60,
    "window_seconds": 60
  },
  "error": "Rate limit exceeded",
  "error_code": "RATE_LIMIT_EXCEEDED"
}

{
  "compliance_analysis": null,
  "documents": [
    {
      "doc_metadata": {
        "compliance_status": "noncompliant",
        "coverage": "general",
        "date_of_issue": "2024-05-23",
        "is_approved": true,
        "revised_date": "2024-05-01",
        "version": "RoHS III"
      },
      "level": "Component",
      "type": "RoHS",
      "uploaded_at": "2025-12-30T11:49:58.074001",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Company",
      "type": "RoHS Exempt",
      "uploaded_at": "2025-12-05T11:49:58.074016",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf"
    },
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2025-09-02",
        "is_approved": true,
        "revised_date": "2025-08-26",
        "version": "REACH"
      },
      "level": "Company",
      "type": "REACH",
      "uploaded_at": "2025-12-05T11:49:58.074023",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Series",
      "type": "Datasheet",
      "uploaded_at": "2025-12-05T11:49:58.308975",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf"
    }
  ],
  "internal_id": "TI LM 99-358 - QX",
  "manufacturer": "Texas Instruments",
  "mpn": "LM358",
  "processing_time_ms": 0,
  "product_url": "https://www.ti.com/product/TILM99-358-QX",
  "status": "active"
}

#26 Failed 504

2025-12-06 11:10:15 (36.95s)

AI: The test case resulted in a 'stream timeout' and a 504 Gateway Timeout status, indicating a critical failure to retrieve the expected product and compliance data.

Expected status 200, got 504

AI Functional Analysis

Summary

The test case resulted in a 'stream timeout' and a 504 Gateway Timeout status, indicating a critical failure to retrieve the expected product and compliance data.

Detailed Report

This test case, 'JEF - All Critical - Test', failed significantly. The system returned a 'stream timeout' message and an HTTP status code of 504 Gateway Timeout, rather than the expected 200 OK status with the detailed JSON response containing critical product and compliance information. This represents a complete failure to deliver the functional requirements of the test, as no data was returned whatsoever. The observed outcome is unacceptable. A 504 status code typically indicates that a server acting as a gateway or proxy did not receive a timely response from an upstream server it needed to access to complete the request. In this scenario, the user would experience a broken or unresponsive application, unable to retrieve essential data for the specified MPN 'LM358'. The expected response included crucial details like RoHS and REACH declarations, compliance statuses, and document URLs, all of which are vital for supply chain and regulatory compliance, making this a high-impact failure. Further investigation is immediately required to diagnose the root cause of the timeout. Potential causes include, but are not limited to, backend service performance issues, database query inefficiencies, network latency, or an overloaded server unable to process the request within the configured timeout period. It is critical to identify and resolve the bottleneck to ensure the reliability and responsiveness of the system when retrieving essential compliance documentation.

"stream timeout"

{
  "compliance_analysis": null,
  "documents": [
    {
      "doc_metadata": {
        "compliance_status": "noncompliant",
        "coverage": "general",
        "date_of_issue": "2024-05-23",
        "is_approved": true,
        "revised_date": "2024-05-01",
        "version": "RoHS III"
      },
      "level": "Component",
      "type": "RoHS",
      "uploaded_at": "2025-12-30T11:49:58.074001",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Company",
      "type": "RoHS Exempt",
      "uploaded_at": "2025-12-05T11:49:58.074016",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf"
    },
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2025-09-02",
        "is_approved": true,
        "revised_date": "2025-08-26",
        "version": "REACH"
      },
      "level": "Company",
      "type": "REACH",
      "uploaded_at": "2025-12-05T11:49:58.074023",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Series",
      "type": "Datasheet",
      "uploaded_at": "2025-12-05T11:49:58.308975",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf"
    }
  ],
  "internal_id": "TI LM 99-358 - QX",
  "manufacturer": "Texas Instruments",
  "mpn": "LM358",
  "processing_time_ms": 0,
  "product_url": "https://www.ti.com/product/TILM99-358-QX",
  "status": "active"
}

#5 Failed 200

2025-12-05 11:53:57 (11.27s)

AI: The test failed due to several critical discrepancies, primarily the absence of the `internal_id`, an incorrect `product_url`, and significant differences in the returned RoHS document's level, URL, and compliance status.

Critical differences detected

AI Functional Analysis

Summary

The test failed due to several critical discrepancies, primarily the absence of the `internal_id`, an incorrect `product_url`, and significant differences in the returned RoHS document's level, URL, and compliance status.

Detailed Report

The test case 'JEF - All Critical - Test' resulted in a failure with critical differences detected. The actual response deviates significantly from the expected, particularly regarding key identification fields and critical compliance information. The `internal_id` field, expected to be 'TI LM 99-358 - QX', was returned as `null`. Consequently, the `product_url` also mismatched, using 'LM358' (the MPN) instead of the expected 'TILM99-358-QX' (derived from the internal ID), indicating a failure to correctly identify and utilize the specific internal part identifier. Furthermore, the details for the first 'RoHS' document presented several critical discrepancies. The `level` was expected to be 'Component' but was returned as 'Company'. More importantly, the `compliance_status` within the RoHS document metadata was 'noncompliant' in the expected response but 'compliant' in the actual response. This is a severe functional deviation, as it provides contradictory compliance information for what appears to be the same type of declaration, potentially referencing a different document as indicated by the differing URL (`rohs_996c12dd.pdf` vs. `rohs_772c12dd.pdf`). The differing `uploaded_at` timestamps for most documents, while minor on their own, also suggest the possibility of different data being retrieved or processed at different times. Given the explicit title 'All Critical - Test', these discrepancies are unacceptable. The missing `internal_id` and the subsequent incorrect `product_url` point to a fundamental issue in retrieving or constructing product-specific identification. The reversal of the RoHS compliance status from 'noncompliant' to 'compliant', coupled with a change in document level and URL, represents a critical data integrity failure that could have significant regulatory and operational implications. This result indicates a need for immediate investigation into the data retrieval, processing logic, and potential source data issues.

{
  "compliance_analysis": null,
  "documents": [
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2024-05-23",
        "is_approved": true,
        "revised_date": "2024-05-01",
        "version": "RoHS III"
      },
      "level": "Company",
      "type": "RoHS",
      "uploaded_at": "2025-12-05T11:50:40.786914",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_772c12dd.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Company",
      "type": "RoHS Exempt",
      "uploaded_at": "2025-12-05T11:50:40.786924",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf"
    },
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2025-09-02",
        "is_approved": true,
        "revised_date": "2025-08-26",
        "version": "REACH"
      },
      "level": "Company",
      "type": "REACH",
      "uploaded_at": "2025-12-05T11:50:40.786929",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Series",
      "type": "Datasheet",
      "uploaded_at": "2025-12-05T11:50:40.829665",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf"
    }
  ],
  "internal_id": null,
  "manufacturer": "Texas Instruments",
  "mpn": "LM358",
  "processing_time_ms": 0,
  "product_url": "https://www.ti.com/product/LM358",
  "status": "active"
}

{
  "compliance_analysis": null,
  "documents": [
    {
      "doc_metadata": {
        "compliance_status": "noncompliant",
        "coverage": "general",
        "date_of_issue": "2024-05-23",
        "is_approved": true,
        "revised_date": "2024-05-01",
        "version": "RoHS III"
      },
      "level": "Component",
      "type": "RoHS",
      "uploaded_at": "2025-12-30T11:49:58.074001",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Company",
      "type": "RoHS Exempt",
      "uploaded_at": "2025-12-05T11:49:58.074016",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf"
    },
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2025-09-02",
        "is_approved": true,
        "revised_date": "2025-08-26",
        "version": "REACH"
      },
      "level": "Company",
      "type": "REACH",
      "uploaded_at": "2025-12-05T11:49:58.074023",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Series",
      "type": "Datasheet",
      "uploaded_at": "2025-12-05T11:49:58.308975",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf"
    }
  ],
  "internal_id": "TI LM 99-358 - QX",
  "manufacturer": "Texas Instruments",
  "mpn": "LM358",
  "processing_time_ms": 0,
  "product_url": "https://www.ti.com/product/TILM99-358-QX",
  "status": "active"
}

Response Differences

Total: 1 differences

Path	Expected	Actual	Issue	Severity
documents[0].url	https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf	https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_772c12dd.pdf	value mismatch	Critical

Endpoint Details

Method: POST
Parameter Mode: json
Endpoint: /api/v1/compliance/search
Expected Status

Request Payload

{
  "manufacturer": "texas_instruments",
  "mpn": "LM358"
}

Expected Response

{
  "compliance_analysis": null,
  "documents": [
    {
      "doc_metadata": {
        "compliance_status": "noncompliant",
        "coverage": "general",
        "date_of_issue": "2024-05-23",
        "is_approved": true,
        "revised_date": "2024-05-01",
        "version": "RoHS III"
      },
      "level": "Component",
      "type": "RoHS",
      "uploaded_at": "2025-12-30T11:49:58.074001",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Company",
      "type": "RoHS Exempt",
      "uploaded_at": "2025-12-05T11:49:58.074016",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf"
    },
    {
      "doc_metadata": {
        "compliance_status": "compliant",
        "coverage": "general",
        "date_of_issue": "2025-09-02",
        "is_approved": true,
        "revised_date": "2025-08-26",
        "version": "REACH"
      },
      "level": "Company",
      "type": "REACH",
      "uploaded_at": "2025-12-05T11:49:58.074023",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf"
    },
    {
      "doc_metadata": null,
      "level": "Series",
      "type": "Datasheet",
      "uploaded_at": "2025-12-05T11:49:58.308975",
      "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf"
    }
  ],
  "internal_id": "TI LM 99-358 - QX",
  "manufacturer": "Texas Instruments",
  "mpn": "LM358",
  "processing_time_ms": 0,
  "product_url": "https://www.ti.com/product/TILM99-358-QX",
  "status": "active"
}

Validation Rules

Path	Type	Sample	Status
root	dict	{ "internal_id": "TI LM 99-358 - QX", "mpn": "LM358", "manufacturer": "Texas Instruments", "status": "active", "product_url": "https://www.ti.com/product/TILM99-358-QX", "documents": [ { "type": "RoHS", "level": "Component", "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf", "uploaded_at": "2025-12-30T11:49:58.074001", "doc_metadata": { "is_approved": true, "date_of_issue": "2024-05-23",	Critical
internal_id	str	TI LM 99-358 - QX	Not Important
mpn	str	LM358	Not Important
manufacturer	str	Texas Instruments	Not Important
status	str	active	Critical
product_url	str	https://www.ti.com/product/TILM99-358-QX	Not Important
documents	list	[ { "type": "RoHS", "level": "Component", "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf", "uploaded_at": "2025-12-30T11:49:58.074001", "doc_metadata": { "is_approved": true, "date_of_issue": "2024-05-23", "revised_date": "2024-05-01", "version": "RoHS III", "coverage": "general", "compliance_status": "noncompliant" } }, { "type": "RoHS Exempt", "level": "Company", "url":	Critical
documents[0]	dict	{ "type": "RoHS", "level": "Component", "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf", "uploaded_at": "2025-12-30T11:49:58.074001", "doc_metadata": { "is_approved": true, "date_of_issue": "2024-05-23", "revised_date": "2024-05-01", "version": "RoHS III", "coverage": "general", "compliance_status": "noncompliant" } }	Not Important
documents[0].type	str	RoHS	Not Important
documents[0].level	str	Component	Not Important
documents[0].url	str	https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_996c12dd.pdf	Critical
documents[0].uploaded_at	str	2025-12-30T11:49:58.074001	Not Important
documents[0].doc_metadata	dict	{ "is_approved": true, "date_of_issue": "2024-05-23", "revised_date": "2024-05-01", "version": "RoHS III", "coverage": "general", "compliance_status": "noncompliant" }	Not Important
documents[0].doc_metadata.is_approved	bool	True	Not Important
documents[0].doc_metadata.date_of_issue	str	2024-05-23	Not Important
documents[0].doc_metadata.revised_date	str	2024-05-01	Not Important
documents[0].doc_metadata.version	str	RoHS III	Not Important
documents[0].doc_metadata.coverage	str	general	Not Important
documents[0].doc_metadata.compliance_status	str	noncompliant	Not Important
documents[1]	dict	{ "type": "RoHS Exempt", "level": "Company", "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf", "uploaded_at": "2025-12-05T11:49:58.074016", "doc_metadata": null }	Not Important
documents[1].type	str	RoHS Exempt	Not Important
documents[1].level	str	Company	Not Important
documents[1].url	str	https://beta.certain.li/doc/declarations/texas_instruments/_company/rohs_exempt_b5c49895.pdf	Not Important
documents[1].uploaded_at	str	2025-12-05T11:49:58.074016	Not Important
documents[1].doc_metadata	None	None	Not Important
documents[2]	dict	{ "type": "REACH", "level": "Company", "url": "https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf", "uploaded_at": "2025-12-05T11:49:58.074023", "doc_metadata": { "is_approved": true, "date_of_issue": "2025-09-02", "revised_date": "2025-08-26", "version": "REACH", "coverage": "general", "compliance_status": "compliant" } }	Not Important
documents[2].type	str	REACH	Not Important
documents[2].level	str	Company	Not Important
documents[2].url	str	https://beta.certain.li/doc/declarations/texas_instruments/_company/reach_eda2c207.pdf	Not Important
documents[2].uploaded_at	str	2025-12-05T11:49:58.074023	Not Important
documents[2].doc_metadata	dict	{ "is_approved": true, "date_of_issue": "2025-09-02", "revised_date": "2025-08-26", "version": "REACH", "coverage": "general", "compliance_status": "compliant" }	Not Important
documents[2].doc_metadata.is_approved	bool	True	Not Important
documents[2].doc_metadata.date_of_issue	str	2025-09-02	Not Important
documents[2].doc_metadata.revised_date	str	2025-08-26	Not Important
documents[2].doc_metadata.version	str	REACH	Not Important
documents[2].doc_metadata.coverage	str	general	Not Important
documents[2].doc_metadata.compliance_status	str	compliant	Not Important
documents[3]	dict	{ "type": "Datasheet", "level": "Series", "url": "https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf", "uploaded_at": "2025-12-05T11:49:58.308975", "doc_metadata": null }	Not Important
documents[3].type	str	Datasheet	Not Important
documents[3].level	str	Series	Not Important
documents[3].url	str	https://beta.certain.li/doc/declarations/texas_instruments/LM358/datasheet_2735fe86.pdf	Not Important
documents[3].uploaded_at	str	2025-12-05T11:49:58.308975	Not Important
documents[3].doc_metadata	None	None	Not Important
compliance_analysis	None	None	Not Important
processing_time_ms	int	0	Not Important

JEF - All Critical - Test

Test Results

#66 Failed 429

AI Functional Analysis

Summary

Detailed Report

Visual Difference

#46 Failed 429

AI Functional Analysis

Summary

Detailed Report

Visual Difference

#26 Failed 504

AI Functional Analysis

Summary

Detailed Report

Visual Difference

#5 Failed 200

AI Functional Analysis

Summary

Detailed Report

Visual Difference

Response Differences

Endpoint Details

Request Payload

Expected Response

Validation Rules