Pagination Shows Duplicate Items
When navigating between pages of search results, the same item appears on more than one page. This usually happens because the list is sorted by a non-unique column with no stable secondary sort key — two records tied on the primary sort column can swap positions between page requests, causing one of them to appear at the boundary of consecutive pages.
MediumIntermediateManual testingAPI testingExploratory testing
// UNDERSTAND
// Symptoms
- Product ID 55 ('Acme Widget') appears both at the bottom of page 1 and the top of page 2 of GET /api/products?page=1&limit=10&sort=name
- Paginating through all pages and collecting IDs produces a list with duplicate entries
- The total item count reported by the API is correct, but summing items across all pages yields more records than expected
- Items appear to shift between page boundaries each time the same page is requested
- Sorting by a column with many ties (e.g. category or status) produces the most frequent duplicates
// Root Cause
- The query sorts by a non-unique column (name) with no stable secondary sort key. When two products share the same name, the database engine returns them in an arbitrary order that can differ between page requests. A product at the boundary of two pages can appear on whichever page its arbitrary position lands at in each query.
- The OFFSET/LIMIT pagination strategy lacks a stable sort: between two page requests, a new record inserted at a lower offset shifts all subsequent items down by one position, causing the last item of page N to reappear as the first item of page N+1.
// Where It Appears
- Product catalogues and e-commerce search results sorted by name or category
- Admin dashboards with paginated user or order lists sorted by status
- Content management systems with paginated article lists sorted by publication date
- Any API endpoint that uses OFFSET/LIMIT pagination without a guaranteed unique sort column
// REPRODUCE & TEST
// How to Reproduce
- 01Send GET /api/products?page=1&limit=10&sort=name and record the product IDs returned in the response (e.g. IDs 1–10 including product ID 55 at position 10)
- 02Send GET /api/products?page=2&limit=10&sort=name and record the product IDs returned
- 03Compare the two ID lists; product ID 55 ('Acme Widget') should appear on only one page
- 04If the same product ID appears in both responses, the duplicate-items bug is confirmed
- 05Repeat with sort=category (a column with more ties) to observe a higher frequency of duplicates
// Test Data Needed
- A dataset with at least 20 products, including two or more products that share the same name value (e.g. both named 'Acme Widget') to trigger the non-unique sort collision
- A way to send paginated API requests (Postman, curl, or browser DevTools)
// Manual Testing Ideas
- Page through the full results from page 1 to the last page; collect all product IDs and verify the list contains no duplicates
- Sort by each available column (name, category, status, price) and repeat the multi-page ID collection to find which sort columns trigger duplicates
- While browsing page 1, add a new product in a separate tab, then navigate to page 2 and check whether any products were shifted off the boundary
- Compare the item count returned across all pages against the totalCount value in the API response — a mismatch indicates shifted or duplicated items
- Confirm that sorting by a unique column (e.g. id, created_at DESC) eliminates the duplicates — this confirms the root cause
// API Testing Ideas
- Send GET /api/products?page=1&limit=10&sort=name; record all product IDs in the items array (call this Set A)
- Send GET /api/products?page=2&limit=10&sort=name; record all product IDs in the items array (call this Set B)
- Assert the intersection of Set A and Set B is empty — any shared ID is a duplicate
- Repeat for all subsequent pages until the last page is reached
- Also send GET /api/products?page=1&limit=10&sort=id (unique column) and verify no duplicates appear across pages — this confirms that adding a unique secondary sort key would fix the issue
// Automation Idea
Write a parameterized API test that fetches all pages of GET /api/products?limit=10&sort=name, collects every product ID across all pages into a flat array, and asserts the array contains no duplicate IDs. Assert that len(ids) == len(set(ids)). Extend the test to run with sort=name, sort=category, and sort=id to confirm duplicates occur only on non-unique sort columns.
// Expected Result
Each product appears exactly once across all pages of paginated results, regardless of the sort column used.
// Actual Result (Example)
GET /api/products?page=1&limit=10&sort=name returns product ID 55 ('Acme Widget') at position 10. GET /api/products?page=2&limit=10&sort=name returns product ID 55 ('Acme Widget') at position 1. The same item appears on both pages.
// REPORT IT
Example Bug Report
- Title
- Product ID 55 ('Acme Widget') appears on both page 1 and page 2 of /api/products?sort=name
- Severity
- Medium
- Environment
- Staging environment Postman Valid bearer token 20+ products in database, two products named 'Acme Widget'
- Steps to Reproduce
- 01Send GET /api/products?page=1&limit=10&sort=name with a valid bearer token
- 02Record all product IDs in the response (note product ID 55 at position 10)
- 03Send GET /api/products?page=2&limit=10&sort=name with the same token
- 04Record all product IDs in the response
- 05Compare the two ID lists for any shared values
- Expected Result
- No product ID appears on more than one page.
- Actual Result
- Product ID 55 ('Acme Widget') appears at position 10 of page 1 and position 1 of page 2. The sort by name is non-deterministic for products sharing the same name, causing the item to appear at the page boundary in both responses.
- Impact
- Users browsing all pages of results see duplicate items, creating confusion. Consumers who collect all pages to process (e.g. export, import, batch jobs) process the same record twice, causing data integrity issues downstream.