Solving Magento

Solutions for Magento E-Commerce Platform

by Oleg Ishenko

18

Feb

Magento URL Rewrites

Magento is a web application implementing a Front Controller pattern, and this means that most HTTP requests are routed to the index.php file in the root folder. In order to display a category, a product, or a non-catalog information page Magento parses the incoming URL to determine which modules and controllers must be engaged to process the request. When the URL rewriting is not enabled Magento URLs look like this: catalog/product/view/id/166 or catalog/category/view/id/10. These are valid URLs and they contain important information, which Magento uses to produce HTML output. Thus, catalog points to the module Mage_Catalog, product is translated into ProductControler, view – into viewAction, and the parameter id passes its value “10”. Altogether such URL generates a call to function Mage_Catalog_ProductContoller::viewAction(), which renders a details page for a product with ID 10.

Despite being logical and well fit for an MVC web application which is Magento, this URL pattern has two significant drawbacks. First, it is not human friendly. The IDs used in these URLs tell users nothing about the shop products or categories. Second, these URLs are not search engine optimized because they contain only generic terms (“catalog”, “product”, “category”) that have little effect in achieving high search engine positions. Ranking algorithms of the modern search engines (Google, above all) place high value into terms contained within URLs. For these two reasons, human and search engine compatibility, it is imperative to include product and category names into shop URLs. Descriptive URLs consisting of words meaningful for both humans and search engines improve user experience and place the links to your shop higher in the search results relevant to your business.

Magento uses a rewrite engine to match descriptive URLs to its resources. The rewrite engine is a part of Magento core system and is responsible for matching incoming descriptive URLs to controllers, actions, and entity IDs. It is also tasked with automated creation of descriptive URLs for Magento resources.

URL rewrites in the shop: products and categories

Magento allows rewriting any URL (with one exception discussed below), and offers automated functionality for rewriting category and product URLs. To create descriptive URLs for categories and products the system uses their urlkey attribute. A urlkey value can be either provided explicitly when a catalog entity is created, or generated using the name of the entity. Some names, however, can contain white spaces or other special characters, which may clutter URLs and make them unreadable. Magento doesn’t use product and category names verbatim for URLs. Instead, names are formatted using regular expressions to produce urlkey values:

For products, Mage_Catalog_Model_Product_Url::formatUrlKey($str) :

public function formatUrlKey($str)
{
    $urlKey = preg_replace('#[^0-9a-z]+#i', '-', Mage::helper('catalog/product_url')->format($str));
    $urlKey = strtolower($urlKey);
    $urlKey = trim($urlKey, '-');

    return $urlKey;
}

For categories, Mage_Catalog_Model_Category::formatUrlKey($str) :

public function formatUrlKey($str)
{
    $str = Mage::helper('core')->removeAccents($str);
    $urlKey = preg_replace('#[^0-9a-z]+#i', '-', $str);
    $urlKey = strtolower($urlKey);
    $urlKey = trim($urlKey, '-');
    return $urlKey;
}

When saving products or categories you can provide your own value for urlkey. If not, entity names will be used to generate urlkey as described above.

As I’ve already mentioned, the catalog URL rewrites generation is automated. “Catalog URL Rewrites” is one of the Magento’s indexes, catalog_url, and when set to update “on save” it will react to catalog entities’ being changed and saved. The task of rewrites refresh is performed by the catalog URL model Mage_Catalog_Model_Url. According to its source code, an approximate process of creating of URL rewrites for a category, which is implemented in function Mage_Catalog_Model_Url::refreshCategoryRewrite, is as follows:

  1. Get the ID of a store in which rewrites are generated. If no store information is provided repeat the next steps for every store.
  2. Find out if the current category has subcategories and generate an array containing the IDs of the category and all of its children.
  3. For each category generate rewrite data consisting of id_path, target_path, request_path, store_id, category_id, is_system (more on these properties below).
  4. Create a new rewrite if no rewrite entry with the same combination of store_id, id_path, and is_system exists, or update an existing one.
  5. Check if the category URL has changed and if the store settings mandate creating a permanent redirect from the old URL to the new one. It may happen that the category name is altered. The new urlkey is generated, and with it a new category URL. But the old URL can still exist in search engine caches, on a partner website, or in customer bookmarks. Normally you’d want the old URL still to show the category page instead of a 404 Not Found error. In this case the system will convert old URL rewrites into redirects: a special kind of URL rewrite. The old URLs will be matched to these rewrites and redirected to the new URLs. The redirect is permanent – the HTTP code is 301 as prescribed by SEO best practices for such cases. The redirect rewrites also have the is_system property set to 0, which makes Magento ignore them when updating URL rewrites next time, thus leaving the redirect permanently in the system.
  6. If a product was added to or deleted from the category, refresh the URL rewrites of the child products.
  7. Perform the above steps for the child categories if any.

Product rewrites are generated by function Mage_Catalog_Model_Url::refreshProductRewrite :

  1. Just as with categories, the system decides either to update rewrites for the provided store ID only, or for every store if no store ID is specified.
  2. Find out to which categories the product belongs to, and generate paths out of category urlkey properties.
  3. Save new or update existing URL rewrites for the product itself (without a category path, e.g. ottoman.html) and for every category it belongs to including category paths (e.g., furniture/ottoman.html or sale/ottoman.html).
  4. Check if the product URL has changed, and, if necessary, create 301 redirects.
  5. Remove all older invalid URL rewrites for the product.

It is important to describe the data that is saved in a rewrite entry.

  • id_path serves as an identifier to the rewrite entry and is unique within a store. Its value is comprised of ‘category’ or ‘product’ and the ID of the current category or product, e.g. “category/10” or “product/99”. Products can have multiple rewrites, each per category it is assigned to; and in such cases id_path can be extended with a category id, e.g. “product/99/10”. The id_path property plays a part when a category or a product URL is created – the system generates an id_path and looks into the rewrite table for a matching entry, fetches the request_path and produces a URL.
  • target_path – this property looks like a non-descriptive URL we discussed above. It contains references to the module, controller, action, and entity ID. When dispatching a request, Magento uses the request_path extracted from the request URL to match to a target_path. The matched entry will be used to pass the request to the proper controller action which will render the output.
  • request_path – is a descriptive URL slug generated from the entity’s urlkey attribute. A request_path of a category contains the urlkey of the parent categories, e.g.: furniture/living-room.html for the “living-room” category. Note the ”.html” suffix, which you can control with the configuration settings. For products Magento generates multiple entries with different request_paths. First, a rewrite with a simple request_path consisting only of the urlkey of the product itself, like ottoman.html. Then for every parent directory a category path is added, e.g. furniture/living-room/ottoman.html. The request_path property of sub-categories include a complete path, e.g. furniture/living-room.html.
  • store_id – in a multistore environment there can be multiple rewrites for the same category or product reflecting their possibly different names (internationalization) or category membership.
  • is_system – this property indicates that the rewrite has been created by the system automatically and can be modified when URL rewrites re-indexing starts. Permanent redirects created automatically by the shop when a product or a category urlkey is changed are set to non-system (i.e. is_system = 0) so that they can’t be affected by the URL re-indexing.
  • options – contain a redirect code for the redirect rewrites. Regular URL rewrite entries have nothing in this property – they do not make web clients redirect. But as we have mentioned above, Magento can create redirect rewrites when necessary. These entries will have “PR” (“permanent redirect”) code saved in this property. Shop admins can create custom rewrites and provide redirect options for them. Custom rewrites can be temporary redirects, their options property will have “R” (“redirect”). We will discuss redirect options in details further below.

URL rewrites configuration settings

You can control some important aspects of URL rewrites with the shop configuration. For instance, under Catalog => Search Engine Optimizations you can find the following settings:

  • Product URL Suffix and Category URL Suffix – here you can add a suffix, which will be appended to the product or category urlkey. If you wish your product and category URLs to emulate static HTML files, you can enter ”.html” into these two fields. Then your product URL will be like /nokia-2610-phone.html and category URL like /cell-phones.html. If you leave them empty then your URLs will consist of the urlkey only: /nokia-2610-phone and /cell-phones (see the Best Practices section below for some advice on the URL structure).
  • Use Categories Path for Product URLs – if set to yes, products URLs in category lists will include a category path, like electronics/cell-phones/sony-ericsson-w810i.html. If this setting is disabled then product URLs will only contain a urlkey and, optionally, a suffix: /sony-ericsson-w810i.html.
  • Create Permanent Redirect for URLs if URL Key Changed – whenever you change the urlkey property of a category or a product (either explicitly or by changing the entity’s name, which will generate a new urlkey), new request_path values are created. This means that URLs based on the old request_path will become inaccessible and return a 404 “Not Found” error. To avoid this Magento offers an automated generation of redirects from the old URLs to the new ones.

Another interesting setting can be found under General => Web => Search Engine Optimization. If Use Web Server Rewrites is set to no, it makes Magento extend its base URL with “index.php”, such as www.yoursite.com/index.php/furniture. Despite looking somewhat ugly, this URL pattern is necessary if your server for some reason has the Apache’s mod_rewrite disabled. As you’ve learned from the Front Controller post every HTTP request, for which no file or directory can be located on the server, is routed to the index.php file. This routing is made possible by rewriting rules defined in the root .htaccess file. But these rules have effect only if the mod_rewrite module is enabled. Otherwise such requests will return “404 Not Found” errors. In most cases a Magento shop expects that the mod_rewrite is present and operates with this Use Web Server Rewrites activated. But once this setting is disabled, Magento’s URL will point to index.php and the shop can function without relying on the mod_rewrite rules.

Creating URL rewrites manually, custom URL rewrites

In addition to the automated generation of URL rewrites Magento offers a way to do it manually in the administration back-end interface. You can find it under menu Catalog => URL Rewrite Management. On this page you are presented with a list of all existing URL redirects. You can edit them by clicking on the grid entries. Editing options are restrictive. For system rewrites you can change the redirect option and the request path only. For custom rewrites you can also change the store option. Besides that in both cases you can enter a rewrite description. You can’t change the rewrite type, “custom” to “system” and vice versa. Also, if you manually change the request_path property for a system rewrite, this alteration will be reverted the next time Magento refreshes the Catalog URL Rewrites index.

This interface allows you as a shop administrator to create new URL rewrites manually. You can create three type of rewrites:

  1. Category rewrite. After clicking on the “Add URL Rewrite” button you are shown a tree of the shop categories. Choose one and you are redirected to the second step where you can set the store, request_path, and redirect option property. The property target_path is filled automatically and you can’t modify it.
  2. Product rewrite. In the first step you select “For product” in the drop-down list, after which a product grid is presented to you. You can search for a product or pick one already displayed. In the next step a category tree is shown, where you can pick a category to which the product is assigned. This will add the category path to the rewrite’s request_path and target_path properties. Or you can skip this step, and no category path will be used for the new rewrite. The last step is the same as with a category rewrite: you can modify store, request_path, redirect option, and add a description.
  3. Custom URL rewrite. There is only one step where you provide values for the store, request_path, target_path, and redirect options. Additionally you have to enter an id_path – this value must be unique within the selected store. The request and target paths are of your choosing. For instance, you may provide an alternative URL for the your “Furniture” category by entering “furniture-january-sale.html” into the request_path field and “furniture.html” into the target_path. Or you can make your contact form accessible via “marketing-contact.html” URL (request_path = “marketing-contact.html”, target_path = “contacts/”). Choosing “No” for the redirect option will make the browser display your custom URL in the address field. Choosing “Temporary” or “Permanent” will redirect the browser to the URL in the target_path.

Temporary and Permanent Redirects

The terms “Temporary” and “Permanent” redirects have already been mentioned a few times. But I think, a short summary is necessary. Magento allows these two redirect types and performs them as prescribed in the RFC1945. In both cases the browser changes the URL in its address field to the one it is redirected to. This stands in contrast to the non-redirect rewrites, when Magento displays the content generated by calling the target_path that was matched to the requested URL, which the browser displays in its address field.

Permanent redirects, or 301 “Moved Permanently”, tell web clients that the requested URL is now available at the new address and in future requests the new URL must be used. This makes the 301 redirects especially useful when a category or a product name changes. Web clients using the older URL will be instructed that the target is still available, albeit by a new address. Search engines will make note of that and update their indices accordingly. This will prevent your shop product and category pages from losing their positions in search results.

For these reasons Magento automatically generates 301 redirect rewrites if product or category names change and if your shop’s setting Create Permanent Redirect for URLs if URL Key Changed is set to yes.

You can also create custom 301 redirects, for example, for non-catalog pages. Such rewrite entries will have their options property set to “PR” (“permanent redirect”).

Temporary redirects, or 302 “Moved Temporarily”, tell web clients that the resource is still available, but under a new URL, and this new URL can be changed in the future. For future requests web clients must rely on the original URL. A typical scenario for a temporary redirect is a seasonal sale page, like Easter or Christmas specials, which is normally available only few weeks a year. When not in the season, you can redirect its URL temporarily to the front page or to your regular sale page. Magento does not automatically generate temporary redirect rewrites, but you can create them manually or programmatically. These rewrites will have “R” (“redirect”) in their options property.

Limitations of URL rewrites

The most common oversight with URL rewrites is an attempt to use them for static files. This simply will not work. When a request for a static file is processed, the Magento rewrite engine is not even engaged. So it is defined by the rules in the .htaccess file: if a requested file or directory exists on the server – return it directly. Magento URL rewriting requires that the request is first routed to the index.php file, which starts Magento and provides access to its functionality including URL rewrites. Therefore, creating URL rewrites for static files, i.e. catalog images and other content of the media folder, CSS and JS files, would have no effect.

URL rewrite process flow

As we have discussed in the Front Controller post, incoming requests are dispatched to a proper controller class by the Front Controller object. In function Mage_Core_Controller_Varien_Front::dispatch we find the following line:

Mage::getModel('core/url_rewrite')->rewrite();

Here Magento creates an instance of class Mage_Core_Model_Url_Rewrite and calls its function rewrite . This function uses the request object to extract URL information and to locate a URL rewrite entry. The target path of the URL rewrite is then used to either dispatch the request to the proper controller or to redirect the web client to the target URL. The approximate process is as follows:

  1. Find out if the pathInfo property of the request contains a trailing slash. The pathInfo is the “path” part of the URL, e.g. if the URL is http://yoursite.com/electronics/cell-phones.html?color=24 the path is /electronics/cell-phones.html. In this example the pathInfo has no trailing slash. An opposite would be true for a URL like http://yoursite.com/electronics/cell-phones/.
  2. If no trailing slash found set the variable $altSlash to “/”, if a trailing slash is present, set $altSlash to empty. This “alternative” slash variable will be used later to find a matching URL with or without a trailing slash.
  3. Generate “request cases” – an array of URL paths, with trailing slash, without a trailing slash, with query string or without query string.
  4. Perform a database query against the table core_url_rewrite and fetch all entries whose field ``request_path matches one of the generated “request cases”.
  5. Since there can be multiple results, choose one with the lowest penalty. Penalty is calculated as an integer by using the position of the matched request_path in the “request cases” array. This array is built in the following order: 1. path + query string, path + alternative slash _ query string, path, path + alternative slash. So at the top are values that are closest to the original incoming URL. Values extended with an alternative slash further in the array and will get higher penalty. Also penalty is added to the found entries without a store ID value. Long story short: preferred are rewrites that have the same query string and trailing slash availability as the incoming URL, and are store-specific.
  6. If no matching URL rewrite is found, the request object is not modified and will be processed by the Front Controller as is.
  7. If a matching entry is found, the system checks, if its target path is external, i.e. contains a full URL with a scheme part: “http” or “https”. In this case Magento redirects the web client to this URL.
  8. If the matching entry contains a redirects option, i.e. its options property has “PR” or “R” value, Magento will perform either a permanent or a temporary redirect to this target path.
  9. If the matching entry has no redirect option, Magento will modify the request object by replacing its properties targetUrl and targetPath with values generated from the matched entry’s target_path property.
  10. The control is returned to the Front Controller’s dispatch function. The request properties targetUrl and targetPath are now in form “catalog/controller/action/param1/value1/”, which will allow the Front Controller to dispatch the request to the proper controller class.

The rewrite process flow:

Magento URL rewrite process flow

Best Practices of URL Design

With URL rewrites you have a powerful tool to control your URLs. When designing a URL structure it is important to find a proper balance between the SEO needs and the human readability. Here are a few points you should consider.

  1. Reflecting the data hierarchy. The catalog data is usually arranged into a tree of categories with products as leaf nodes. It may serve well for both search engines and customers to reflect this structure in your URLs. Category URLs in Magento are hierarchical by default. With product URLs you have two choices. Activate the Use Categories Path for Product URLs to add a category path to every product URL so that it displays the product position in the catalog data hierarchy. Or you can leave the category path out of product URLs. One reason to do the latter is when category names are too generic and do not add informational value to URLs.
  2. Using file extensions. Many websites disguise their dynamic URLs as names of static files, usually by adding a ”.html” suffix. It is sometimes done out of belief that search engine rank static file content higher than dynamic, which could have been true a few years ago but is highly debatable now. As long as you are not adding ”.exe” or ”.dll” to your URLs you are free to use any file extension, or none at all. In fact, file extensions are redundant – your URLs point to data, not to files.
  3. Trailing slashes. A trailing slash at the end of a category URL is preferred while product URLs are better left without one. This distinguishes a category resource, which can have children (sub-categories or products), from a leaf node – a product – which has no children nodes.

64 thoughts on “Magento URL Rewrites”

  1. Thanks a lot for the explanation, Oleg, very useful. We have had some problems with the automatic generation of URLs in our site and your article has been very clarifier.
    Our URL Rewrite table usually has a lot of items (around 40.000) and we are worry about this size… Do you think we have to regenerate our URL Rewrite table every some time? Thank you in advance!

    Reply ↓

    • Hi Peter, thank you for your kind words. The problem with duplicate url_keys should be taken care of automatically when the rewrites are refreshed. The code in function Mage_Catalog_Model_Url::getProductRequestPath($product, $category) checks, if the current request path (generated from a url_key) already exists, and if yes, uses the request path extended with a product id, e.g, 30-flat-panel-tft-lcd-cinema-hd-monitor-156.html instead of 30-flat-panel-tft-lcd-cinema-hd-monitor.html. And if the extended path also exists, a unique request path is generated, see function getUnusedPath in the same class. The request path must be unique in the store score, and there is a unique constraint in table core_url_rewrite on columns `request_path` and `store_id`

      Reply ↓

  5. Hi buddy,
    Thanks for the great in-depth article!
    What about Unicode URLs? Even if you can save theme, Magento returns 404 error upon loading the URL.
    Which part of comparison mainly conflicts with the Unicode values?

    Reply ↓

    • The problem is that when you create such a rewrite its request_path gets written to database as is, i.e. a Unicode string. The request to this URL comes in a URL-encoded form, that is “%D9%86%D8%A7%D9%85%20%D9%85%D8%AD%D8%B5%D9%88%D9%84” instead of “نام محصول”. When the database is searched for the encoded string nothing is found, hence the 404 error.

      A solution to this problem could be writing an extension, or a custom router that would run the urldecode PHP function for the incoming request path.

      The core code where the URL-encoded request string is passed to the rewrite procedure is this:

              $pathInfo = $request->getPathInfo();
        $origSlash = (substr($pathInfo, -1) == '/') ? '/' : '';
        $requestPath = trim($pathInfo, '/');

      /app/code/core/Mage/Core/Model/Url/Rewrite.php line 216

      What is necessary to change here (through a model rewrite or a custom router) is to decode the contents of the $requestPath variable:

            $requestPath =urldecode( trim($pathInfo, '/'));
      Reply ↓

  6. Hi Oleg
    Yep its a great in-depth article…
    Im issuing a problem after having truncated the table core_url_rewriting for refreshing the urls (750000 entries VS 12000 prods), as the indexing were never ended.
    I saw in many posts that the solution was to truncate, and Magento will refresh those entries.
    Since then, Magento ver. (1.7.0.2) refuse to reindex this table, with fatal errors, and no entry were re-generated…
    Any Idea whats going on? from where could come this error?
    Thx for your help
    Bap

    Reply ↓

    • Hi Baptiste,

      Your problem had happen to me as well. In my case the reason was the memory limit, which was exhausted by the indexing process. The thing is that the entire URL rewrite index is being built within one MySQL transaction, and no data gets written to the database when the indexer fails. On the other hand the script keeps consuming memory until the limit is reached. We stopped using the back-end to start indexes, instead run them from the command line while specifying more memory than is set for the shop in its .htaccess file. The command we use is:

      php -d memory_limit=2G shell/indexer.php --reindex catalog_url

      This sets the memory limit of 2G to the script, which in our case was sufficient to re-index 10000 products’ URLs.

      Reply ↓

  7. Pingback: Magento .htaccess RewriteRule - custom url redirect for CMS page to external url | Magneto Freelancer

  8. Hi Oleg,
    Thanks for your great article. I’m wondering if custom url rewrites is what would work for my situation…

    I had an oscommerce store, and have changed to Magento. The new Magento store has been setup in a nginx environment, unlike the old site, which was on a standard apache set up (I should point out that I don’t really know much about this, I have a host and a developer, but neither of them can figure this out).

    We have URL redirects set up for most of the old oscommerce site, however we have a few pages that end with “.php” that are causing issues. eg http://www.mysite.com/specials.php and http://www.mysite.com/products_new.php. These addresses are being directed to a very bland “File not found” page (that’s all it says). Host says that “all requests ending in PHP end up hitting Magento, I was not able to intercept the request before it went to the backend, regardless of configuration.”

    We have a development copy of the site (which is on apache), which I used the custom URL option, with the following settings (for example):
    ID Path: specials.php
    Request Path: specials.php
    Target Path: sale.html
    Redirect: Permanent

    These settings work on the development copy, but not on my live copy. Am I doing something wrong, or do you have any suggestions on how to get these pages redirected? At this stage, even just redirecting them to the home page would suffice.

    Thanks,
    Ann

    Reply ↓

    • Hi Ann,

      Sorry for a delay answering your question. If you still haven’t found a solution here are my thoughts on that:

      I presume, specials.php and products_new.php do exist as files. In this case you have the same problem I outlined in section “Limitations of [Magento] URL Rewrites”. Since the files physically present on the server, the .htaccess (or its Nginx equivalent) rewrite configuration directs the request to them directly. The Magento’s index.php script is not engaged, and because of that the custom URL rewrite you defined in the shop’s back-end is not used.

      A solution to that would be defining a URL rewrite in the Nginx configuration that redirects requests to these files to Magento’s index.php.

      Reply ↓

  9. Hi Oleg,

    Thanks for the excelent post!

    Is there any extension or workaround that automatically redirect old urls (long) to new urls (short) after set “Use Categories Path for Product URLs” = yes?

    We have a client that uses long product urls until now, with many urls indexed, and now we want change to short product urls.

    Regards,
    Daniel.

    Reply ↓

    • Hi Daniel,

      Unfortunately I do not know any way to achieve that using Magento configuration only. On the other hand, I don’t think implementing this functionality is all too difficult.

      I also can’t direct you to any third party extension that does that.

      Reply ↓

  10. I did all products by copying existed and editing them. I did not edit Url keys. Now I noticed, that all products are with the same Url key. How it is possible to change automatically Url Keys that they would be done from Product Names.
    Thank you in advance!

    Reply ↓

  11. Hi,
    amazing article!
    thanks for sharing this.
    One question, why does magento need any url rewrite table?
    I mean, this is the source of so many performance issue, so many…
    a simple url rewrite in htaccess + 2 lines of php code to dynamicaly build the URL and that’s it. no DB involved.
    url could be something like
    mysite/prod_whatever_entityId.html
    the php code would generate “whatever” including the product name, the category where the product is displayed etc
    then .htaccess to detect that it is a link to a product (“prod_” keyword) so it extracts the entityId (easy) and redirect to the correct product page.

    I did this with a non-magento website.

    Do you see any rationnal with this?
    I miss the benefit…
    Especially the behavior to maintain the link between the preivous url and the new one when the name changed, so many thinks to manage for nothing?.
    with the solution above, no issue as the “whatever” part does not matter, that’s the key point.
    just the entityId + a keyword to deetect it’s a link to a product (or use “cat_” to mention it is a link to a category.

    I will seriously think about doing this in magento, but I’m new and a bit afraid to break something 🙂

    thanks again
    rod

    Reply ↓

  12. Pingback: custom menu : Magento | ZingWings

  13. Hi Oleg,

    Very nice article indeed.
    Things are explicated in a very teaching way.

    I’m facing a situation on a website with 50k products, 20 attributes and something like … 50 websites on the same serveur/ magento install. (1.7.0.2 CE)

    As you can imagine, the indexing process takes a looooong time…
    Especially the catalog_url one …

    Considering that the products on each websites are the same (among the 50k ones), do you think there could be an easy way to generale urls for the 50k products like “my_cat/my_product_XXX.html”, that would be used by Magento for every 50 websites, without having to insert 50 times the same line in the catalog_url table…

    I hope it’s understandable …

    Thanks for you advice on this 😉

    Regards,
    Fabien

    Reply ↓

    • Hi Fabien,

      Unfortunately I can’t give you a good advice based on your problem’s description. Do you use configurable products? I had a case where the indexing load was decreased by overriding the rewrite methods to exclude configurable children – they were set to be invisible in catalog. But anyway, I am afraid I can’t advise knowing so little about your system.

      Reply ↓

  14. Hi Oleg,

    I am exploring Magento and I am encountering a problem I guess is related to your discussion in this thread. Please see the error message pasted below.

    Any idea of what could be wrong? I have done what was suggested in: http://www.magentocommerce.com/boards%20/viewthread/26166/#t86792 but no success.

    Thanks,

    Wole

    ==========================================
    Object not found!

    The requested URL was not found on this server. The link on the referring page seems to be wrong or outdated. Please inform the author of that page about the error.

    If you think this is a server error, please contact the webmaster.
    Error 404
    ==========================================

    Reply ↓

    • Hi Wole,

      This message is not from Magento, but from the web server. You should check if your .htaccess exists and its content related to the URL rewrites is not altered.

      Reply ↓

  16. Hello,

    I went to URL rewrite.

    Noted the info from the category and deleted all the URL’s in the four stores (English/German/Spanish/Russian).

    Went ahead and did a custom URL re-write to Permanently redirect to my desired URL.

    But they all reverted back to the old system generated URL 🙁

    Can you please let me know what to do to stop the automatic reverting of the URL’s?

    Thanks a million!

    G

    Reply ↓

    • If I understand you correctly, you want the category to be available under a new URL and all the old URLs must redirect to the new one. If that is true, then you should change the category’s url-key attribute to the new URL’s slug and save the category. With the Create Permanent Redirect for old URL option enabled the system will automatically disable old URLs and create redirects to the new one.

      Reply ↓

      • Oleg, you understood correctly, and thank you so much for the reply.

        Everything is fine, until I hit re-index data, at which point the URL gets switched back to the originally assigned URL to the category in question.

        Is there a way for me to stop Magento from doing that when I hit re-index?

        Cheers,

        G

        Reply ↓

        • Sorry for a delayed reply. If the category URL slug is not changed, the old URL rewrites will be automatically restored by the next reindex – that is what you’ve been experiencing. You can’t stop old URLs from being re-created unless you change the category’s “URL key” attribute. The system will take care of redirecting old URLs to the new one and you do not need to create a custom rewrite.

          Reply ↓

          • “You can’t stop old URLs from being re-created unless you change the category’s “URL key” attribute. The system will take care of redirecting old URLs to the new one and you do not need to create a custom rewrite.”

            If I understand correctly, this applies to a strict URL key change. But what about a URL redirect Oleg? For example, I have a blog page (installed blog on the site) and I want to create a category so that it goes into the navigation panel titled “Blog” as well. So I create the “Blog” category and I assign it a URL key (I can’t assign it the same one as the blog on the spot there, because if I do it generates a number at the end, say like /blog-1.html). So….I go to URL permanent redirect and redirect the blog-1 to blog, right. But after re-index it goes back to blog-1.

            Now unless I completely misunderstood what you are trying to say, you’re saying that if I simply put in /blog it would redirect it to the blog page?

            Sorry to be a pain Oleg, but I really have to figure this one out.

          • I am not familiar with the blog module you are using but I suppose it adds an entry to the core_url_rewrite table and that is why you can’t create a category with the same request path – Magento will add an number to it to make it unique just as you described (blog.html -> blog-1.html).

            When re-indexing the system will recreate all rewrites of type “system”. “Custom” rewrites are not affected by reindexing.

  19. Hi Oleg,

    Thank you for this excellent and well thought out article.

    I have the following question:

    If I save a product under category x, and then later remove it from category x and save it under category y, this will have created 2 URL rewrites.

    I understand that for SEO this might be helpful since the original link to my product in category x wont be a dead end, even after the product is removed from category x. This makes sense if the search engines already indexed my site, however if I have accidentally placed items in the incorrect category (i.e. category x), and I immediately remove them, I am stuck with useless URL rewrites. Not a big deal for one product, but this adds up when I move 1000 products around.

    Is there any detrimental effect to simply removing those entries form core_url_rewrite that I don’t want?
    The only easy way I can do this is directly in the db, since mage admin can only delete one entry at a time.

    Thanks.

    Reply ↓

    • Hi Mordy,

      In your case, I think, deleting the unnecessary entries is the simplest way to deal with the problem. Their removal should not have any negative effect on your shop – as you are saying, these have been created by an accident serve no purpose.

      Reply ↓

  20. I am just still confused… need a little help please 🙂

    Are you saying its perfectly fine to have all these old url rewrites for products in categories that are not in those categories any more?

    From 1 old product I have dug through, google doesn’t even have 99% of those urls (product in old categories) in their index… so its probably ok to delete those rewrites?

    I have about 500 products with about 6500+ rewrites… that seems like a bad thing.

    I also think its impossible to manage, I would think I would need to start over.

    Is there a way for magento to delete those old rewrites when products are removed from categories?

    Any help greatly appreciated!!!

    thank you!

    Reply ↓

    • As far as I know, Magento removes all old category-related rewrites when a product is moved to another category, including the “permanent redirect” rewrites.

      The “permanent redirects” are created only when the url_key of a category or a product changes. The number of these “permanent redirects” often gets out of control when category or product names (url_keys) are frequently modified.

      If you are sure that these rewrites are not used by Google, feel free to remove them. Unfortunately, the only way to do that is manually write and execute an SQL query against the core_url_rewrite table.

      Reply ↓

  21. “Use Categories Path for Product URLs”, I don’t know it is how to work… My website set it to “No”, but my product page link like “catelog/view…/id/…”. I search by many way and could not got my want answer. Can you help me?

    Reply ↓

  22. Oleg, very good in-depth post on such a complicated topic!

    I’m struggling at the moment and wondered if you may be able to help shed some light on my little problem.

    When you are running advertising campaigns you need to be able to pass query parameters after the url such as “?source=MyAd”

    The problem i’m having is on product pages as Magento seems to strip all query parameters out of the URL. Is there any htaccess mod that can be done to stop this happening

    Thanks
    Nigel

    Reply ↓

  23. Hi,

    I have created 3 redirects
    1.
    design-your-own/car.html -> catalog/category/view/id/41 (301)
    2.
    catalog/category/view/id/41 ->customcar (301)
    3.
    customcar ->customizer/car/index/?product_no=419(no redirect)

    It redirects fine but i am not getting the query string value product_no.
    Any idea what’s the problem?

    Reply ↓

  24. Hi, this is an excellent article and I’m really happy to have found it. I have a question that troubles me the last couple of days and you are obviously the one to answer it.
    I am trying to set up two stores in magento with different domain names which share the cart. Is there a way to remove the store code (__store=default, __store=store2) from the url when using the store switcher?

    Thanks in advance,
    Panos

    Reply ↓

    • Hi Panos,

      Since your stores have different domain names, you could try creating an .htaccess rule that would recognize the __store query variable and use its value to redirect to a URL with a corresponding domain name

      Reply ↓

  25. Hi Oleg,
    Amazing article. Really well written and clear. We have a problem with our stores which I am hoping you might throw some light on. We have problems because all product urls are being rewritten with numbers at the end. eg http://www.jldev.co.uk/greenworks-2400007-24v-lithium-ion-cordless-battery-leaf-blower-1075.html.

    Is this because the product is in multiple categories? And is there any way to fix? I have tried everything I can think of and all solutions on internet dont work for us!

    Thanks, Alex

    Reply ↓

  27. Hi Oleg,

    Thanks to your very helpful article, I was able to modify Magento so that it strips out trademarks instead of including them in the url key (e.g., ‘™’ => ‘tm’). However, one thing that I can’t figure out is how to change the behavior by overriding instead of modifying the original Magento file. Let me explain: if I modify $_convertTable in docroot/app/code/core/Mage/Catalog/Helper/Product/Url.php, it doesn’t work. I verified that it only works when I modify $_convertTable inside docroot/includes/src/Mage_Catalog_Helper_Product_Url.php.

    I really don’t want to touch anything in the ‘includes’ directory, so how would I be able to make this work by overriding Url.php instead?

    Reply ↓

  28. Hi, Oleg and thank you for this website, such a great and thorough resource for understanding Magento!

    I’m getting a magento shop close to release and I’ve got 64 000+ records on URL Rewrite Management for only 100+ products. That’s bad.

    Now I also realized that there are multiple useless rewrites which have been generated while I’ve been adding simple products by duplicating older products and changing the attributes a bit. I’m sure I’ve always at least changed the name of the product, and usually also erased/modified the duplicated url key and unticked the “Create Permanent Redirect for old URL” box. Nevertheless I see that I’ve got live urls like these:
    My Product –> my-product (ok)
    Another Duplicated product –> my-product-2314 (not at all ok, should be another-duplicated-product).

    And when I try to use the another-duplicated-product URL key in the browser, I only get 404.

    So I thought I’d rather wipe everything clean before launching the site for the public. All the product names are different, so I’d like just to have urls simply generated out of them as Magento automatically does.

    Can I do it just by going to phpmyadmin, open core_url_rewrite table, select all rows and hit delete? (I’m not very comfortable with the command line UI 😉 ) And then I suppose I ought to go to admin > index management and reindex Catalog URL Rewrites?

    Does this help the problem to solve and the number of url rewrites not to get out of the hand later?

    What do you suggest?

    Reply ↓

  29. Pingback: Magento performance – Catalog URL indexing | ionicmelon

  30. I have 43,806 rows under Catalog-> URL Rewrite Management but my sitemap.xml file contains only 2,035 URLs. I need to delete the entries in the core_url_rewrite table in order to get rid of duplicate meta tag issues in Google Web Master due to some of the old URLs we have in the URL rewrites. The problem is the core_url_rewrite is empty and there are no records in it so where are all these URL rewrites stored? How do I get rid of these old URL rewrites? I rfreshed the “Catalog URL Rewrites” index as well but it didn’t help. I am using MAgento 1.7.0.2

    Any ideas?

    Thanks.

    Reply ↓

  31. hi,
    very helpful tutorial.

    can u help me to understand how classes are added in body tag when any product page is loaded.

    Reply ↓

  33. Hi Oleg

    Thank you for writing such a nice article.

    I see some readers mention about querystring getting stripped off on the product page. May I know if this issue has been resolved and if do how as I am having a similar issue on a Magento 1.9.1 CE.

    Would greatly appreciate if you or any of your readers can help.

    Thanks
    Nitin

    Reply ↓

  34. I have an old store based on osCommerce and we are switching to Magento 1.9.2.2. There are some of the old pages that have really good google ranking that we don’t want to ignore and I would like to set up Magento URL Rewrite for these urls. So the old URL (A) will look something like this http://www.website.com/store/index.php?Path=4_4 (an old category page). Another URL (B) may look like this http://www.website.com/store/product_info.php?Path=4_4&id=4 (an old product page).

    I set up the Magento redirect for both URLs following the official instructions and guide lines from Magento as well as this article here. URL B works like a charm. Not a problem. When I enter it, it goes to the new magento product page, with URL 301 rewrite and everything.

    When I type in URL A it rewrites the URL to http://www.website.com/store/?Path=4_4 (so it drops the index.php) and displays the Magento 404 Oops page. So this tells me that the Nginx redirect is working (obviously), but that when it hits Magento it does not seem to like the index.php in that URL? Why? Is there a work around? If anyone knows please advise. Or if anyone knows how I could trouble shoot or log this issue please advise.

    Thank you,

    Martin

    Reply ↓

  35. Thanks for your tutorial. my question i have around 700 store views (250 website each website having three store views) my core_rewrite table is almost 9 gb. one thing is that for all storeview my product , category url is same. so is that possible that i can limit magento that it generate only store_id “0” so it show one url for all stores views. where i can change so while indexing it create only one store id please guide me thanks

    Reply ↓

  36. Hi Oleg,

    Is the any php code to remove the url rewrite?

    Like my url is mywebsite.com/xyz.html and it always give the number infront like mywebsite.com/xyz-1.html

    How do I stop this? Please help me with the php code to not show the rewrites

    Thank you.

    Reply ↓

  37. Hi Oleg,

    Thanks for the information, very helpful!

    I am experiencing another URL issue where parameters are transferred through from a category page, for example, website.com/trousers?limit=48&p=3

    through to the product page, for example, website.com/green-courdroy-trousers?limit=48&p=3

    Is this related to the URL rewrite function?

    Reply ↓

  38. Hi,Oleg. I want to know how to redirect all diabled product to its parent categories. Is there a way I can change codes and make it happen?

    Reply ↓

  39. Hi,

    We have some 45,000 products on our website and the core_url_rewrite table size has already reached 28 lakh plus and it is keep on increasing with every subsequent product upload or if any product is edited (I Guess). This has greatly slowed our website search feature and other functionalities .

    If someone can help us out with this problem i will be very grateful as this issue is highly affecting our business. We don’t have multiple stores and only one store is there

    We are using Magento 1.9.2.2 CE version

    Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *

12 years ago 64 Comments

Magento under the Hood

Theme: Esquire by Matthew Buchanan.