Magento 2 sitemap wrong paths in sitemap index when running on cron

[key-dev]

Preconditions

1. Magento 2.2.4
2. Apache 2.4
3. document root public_html

Steps to reproduce
Add a sitemap with a path of /sitemaps/
Allow cron to generate the sitemap

Expected result
sitemap.xml index urls to sitemap should be https://site.com/sitemaps/sitemap-1-1.xml

Actual result
Sitemap.xml urls look like: https://site.com/public_html/sitemaps/sitemap1-1.xml

If I generate the sitemap through the Magento2 admin, the urls appear as they should.

Moreover after sometime sitemap-1-1.xml gets removed from the sitemaps folder. And never comes back while running on cron.

[magento-engcom-team]

Hi @key-dev. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@key-dev do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[magento-engcom-team]

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• 6. Add label Issue: Confirmed once verification is complete.

• 7. Make sure that automatic system confirms that report has been added to the backlog.

[ghost]

Hi @key-dev thank you for you report, this is acknowledged issue please look ->#5321

[key-dev]

Hi there,
Thanks for updating on it but my issue is different. #5321 is a different issue with image source path. But my issue is with children paths in the sitemap index.
In sitemap.xml all children paths are having public_html in it. We are not using Nginx server we are using apache.
I hope you will consider it as separate issue.
Thanks

[ghost]

HI @key-dev they don't have matter if you use nginx or apache, the problem with pub folders.

[key-dev]

Hi @engcom-backlog-nazar so if my sitemap.xml index file has these sitemaps in it.

https://www.site.com/public_html/sitemaps/sitemap-1-1.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-2.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-3.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-4.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-5.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-6.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-7.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-8.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-9.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-10.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-11.xml2018-12-04T15:44:59+00:00 https://www.site.com/public_html/sitemaps/sitemap-1-12.xml2018-12-04T15:44:59+00:00

Now 2 issues in it
1- Children sitemap has public_html in it. As Magento 2 root is public_html
2- sitemap-1-1.xml gets removed after few hours.
If you could address me to any issue similar to mine would be great.
Thank you

[ask694]

I'm also having this same issue, some times magento auto generating with the wrong paths by adding public_html along with the store url. Please i need a solution for this.

[ghost]

@ask694 this is solution -> #19598

[ask694]

Hi, @engcom-backlog-nazar I can't understant the solution, can you please explain and post here.?

[ghost]

Hi @ask694 You need to add appEmulation in app/code/Magento/Sitemap/Model/Observer.php

<?php
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */
namespace Magento\Sitemap\Model;

use Magento\Store\Model\App\Emulation;
/**
 * Sitemap module observer
 *
 * @author      Magento Core Team <core@magentocommerce.com>
 */
class Observer
{
    /**
     * Enable/disable configuration
     */
    const XML_PATH_GENERATION_ENABLED = 'sitemap/generate/enabled';

    /**
     * Cronjob expression configuration
     *
     * @deprecated Use \Magento\Cron\Model\Config\Backend\Sitemap::CRON_STRING_PATH instead.
     */
    const XML_PATH_CRON_EXPR = 'crontab/default/jobs/generate_sitemaps/schedule/cron_expr';

    /**
     * Error email template configuration
     */
    const XML_PATH_ERROR_TEMPLATE = 'sitemap/generate/error_email_template';

    /**
     * Error email identity configuration
     */
    const XML_PATH_ERROR_IDENTITY = 'sitemap/generate/error_email_identity';

    /**
     * 'Send error emails to' configuration
     */
    const XML_PATH_ERROR_RECIPIENT = 'sitemap/generate/error_email';

    /**
     * Core store config
     *
     * @var \Magento\Framework\App\Config\ScopeConfigInterface
     */
    protected $_scopeConfig;

    /**
     * @var \Magento\Sitemap\Model\ResourceModel\Sitemap\CollectionFactory
     */
    protected $_collectionFactory;

    /**
     * @var \Magento\Framework\Mail\Template\TransportBuilder
     */
    protected $_transportBuilder;

    /**
     * @var \Magento\Store\Model\StoreManagerInterface
     */
    protected $_storeManager;

    /**
     * @var \Magento\Framework\Translate\Inline\StateInterface
     */
    protected $inlineTranslation;

    /**
     * @var Emulation
     */
    private $appEmulation;

    /**
     * @param \Magento\Framework\App\Config\ScopeConfigInterface $scopeConfig
     * @param \Magento\Sitemap\Model\ResourceModel\Sitemap\CollectionFactory $collectionFactory
     * @param \Magento\Store\Model\StoreManagerInterface $storeManager
     * @param \Magento\Framework\Mail\Template\TransportBuilder $transportBuilder
     * @param \Magento\Framework\Translate\Inline\StateInterface $inlineTranslation
     * @param Emulation $appEmulation
     */
    public function __construct(
        \Magento\Framework\App\Config\ScopeConfigInterface $scopeConfig,
        \Magento\Sitemap\Model\ResourceModel\Sitemap\CollectionFactory $collectionFactory,
        \Magento\Store\Model\StoreManagerInterface $storeManager,
        \Magento\Framework\Mail\Template\TransportBuilder $transportBuilder,
        \Magento\Framework\Translate\Inline\StateInterface $inlineTranslation,
        Emulation $appEmulation
    ) {
        $this->_scopeConfig = $scopeConfig;
        $this->_collectionFactory = $collectionFactory;
        $this->_storeManager = $storeManager;
        $this->_transportBuilder = $transportBuilder;
        $this->inlineTranslation = $inlineTranslation;
        $this->appEmulation = $appEmulation;
    }

    /**
     * Generate sitemaps
     *
     * @return void
     * @throws \Exception
     * @SuppressWarnings(PHPMD.UnusedLocalVariable)
     */
    public function scheduledGenerateSitemaps()
    {
        $errors = [];

        // check if scheduled generation enabled
        if (!$this->_scopeConfig->isSetFlag(
            self::XML_PATH_GENERATION_ENABLED,
            \Magento\Store\Model\ScopeInterface::SCOPE_STORE
        )
        ) {
            return;
        }

        $collection = $this->_collectionFactory->create();
        /* @var $collection \Magento\Sitemap\Model\ResourceModel\Sitemap\Collection */
        foreach ($collection as $sitemap) {
            /* @var $sitemap \Magento\Sitemap\Model\Sitemap */
            try {
                $this->appEmulation->startEnvironmentEmulation(
                    $sitemap->getStoreId(),
                    \Magento\Framework\App\Area::AREA_FRONTEND,
                    true
                );
                $sitemap->generateXml();
            } catch (\Exception $e) {
                $errors[] = $e->getMessage();
            } finally {
                $this->appEmulation->stopEnvironmentEmulation();
            }
        }

        if ($errors && $this->_scopeConfig->getValue(
            self::XML_PATH_ERROR_RECIPIENT,
            \Magento\Store\Model\ScopeInterface::SCOPE_STORE
        )
        ) {
            $this->inlineTranslation->suspend();

            $this->_transportBuilder->setTemplateIdentifier(
                $this->_scopeConfig->getValue(
                    self::XML_PATH_ERROR_TEMPLATE,
                    \Magento\Store\Model\ScopeInterface::SCOPE_STORE
                )
            )->setTemplateOptions(
                [
                    'area' => \Magento\Backend\App\Area\FrontNameResolver::AREA_CODE,
                    'store' => \Magento\Store\Model\Store::DEFAULT_STORE_ID,
                ]
            )->setTemplateVars(
                ['warnings' => join("\n", $errors)]
            )->setFrom(
                $this->_scopeConfig->getValue(
                    self::XML_PATH_ERROR_IDENTITY,
                    \Magento\Store\Model\ScopeInterface::SCOPE_STORE
                )
            )->addTo(
                $this->_scopeConfig->getValue(
                    self::XML_PATH_ERROR_RECIPIENT,
                    \Magento\Store\Model\ScopeInterface::SCOPE_STORE
                )
            );
            $transport = $this->_transportBuilder->getTransport();
            $transport->sendMessage();

            $this->inlineTranslation->resume();
        }
    }
}

[ask694]

Hi @magento-engcom-team This fix not worked for me. i had changed the code directly to the core module. Any other solution.? still i'm getting the public_html on the sitemap xml

[ghost]

HI @ask694 you can recheck on 2.3-develop branch

[ghatamehta]

Hi @magento-engcom-team ,

I'm having the same problem with Magento 2.2.3. I have set pub as a document root and my sitemap folder is in html/pub/.

When generated by cron, the sitemap.xml file contains url like this:
http://mysite.com/html/pub/sitemap/sitemap-1-1.xml
http://mysite.com/html/pub/sitemap/sitemap-1-2.xml

But when generated manually through Admin They are :

http://mysite.com/sitemap/sitemap-1-1.xml
http://mysite.com/sitemap/sitemap-1-2.xml

Any solution?

[hostep]

@ghatamehta: as a test, can you try to follow this advice?

adding this line to app/etc/env.php solves the problem
'directories' => [ 'document_root_is_pub' => true ],

And see if that solves your problem?

I haven't tested the above solution yet, but we used to solve this problem by setting the base media url's configuration as follows, which also fixes this problem:

Stores => Configuration => General => Web:

• Base Urls => Base URL for User Media Files => {{unsecure_base_url}}media/
• Base URLs (Secure) => Secure Base URL for User Media Files => {{secure_base_url}}media/

But I'm more in favor of the first solution, it looks cleaner to me.

[ghatamehta]

Hi @hostep ,

Thank you for replying. I already have added these solutions but I have over 100000 pages. So my sitemaps are divided into parts. The sitemap index file contains URLs like

http://mysite.com/html/pub/sitemap/sitemap-1-1.xml
http://mysite.com/html/pub/sitemap/sitemap-1-2.xml

which are not correct.

[frostbitten]

My sitemaps are also experiencing this behavior where they are split into multiple files with the main sitemap.xml incorrectly including "public_html" in the child paths.

But, the problem seems to only occur when I manually update the sitemap through the admin panel. If I use netz98's "magerun" to generate the sitemaps (via bin/n98-magerun2 sys:cron:run sitemap_generate) everything turns out fine. I haven't yet gotten the chance to see what cron does when run via its usual schedule.

Using v 2.3.2