Why does importing fail after creating a module under Python 3 on Windows?










1















The following code tries to create-then-import two modules:



# coding: utf-8

import os
import time

# Remove the modules we're about to create if they already exist
def force_unlink(name):
try:
os.unlink(name)
except OSError:
pass
force_unlink("print1.py")
force_unlink("print1.pyc")
force_unlink("print2.py")
force_unlink("print2.pyc")
time.sleep(1)

# Create module 1 and module 2, then try to import them just afterwards
print("Creating module 1...")
with open("print1.py", "wb+") as fd:
fd.write(b'print("Imported module 1")')
import print1
print("Creating module 2...")
with open("print2.py", "wb+") as fd:
fd.write(b'print("Imported module 2")')
import print2


On Windows, both imports work under Python 2 (2.7), but not under Python 3 (3.5 and 3.6):



$ python2 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Imported module 2
$ python3 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Traceback (most recent call last):
File "reproduce.py", line 26, in <module>
import print2
ImportError: No module named 'print2'


Adding time.sleep(5) before each import printX call makes it work.



Why is that?



Note: This is a simpler version of an issue I'm trying to figure out.










share|improve this question
























  • The line numbers in your posted code don't match the line numbers in your error message. Can you run the version of the code you posted, and post the results of that?

    – user2357112
    Oct 22 '18 at 16:44











  • @user2357112 The original script had coding: utf-8 on top. I added that, but it shouldn't really affect the results.

    – F.X.
    Oct 22 '18 at 16:49







  • 1





    Works fine in 3.7 on windows a well.

    – tgikal
    Oct 22 '18 at 16:51











  • Try generating temporary directories to put the files in, then import those. Or try writing the file into a temporary location, then move it into the final location (renames could trigger a different codepath). Or simpler still: write both files, then import them both after writing. Then generate a third and try to import that, and see if that 3rd module now fails. If that confirms there is a cache, try to see where the cache is located, in Python or on Windows. Use os.listdir(). Use os.scandir(), etc.

    – Martijn Pieters
    Oct 22 '18 at 16:56






  • 1





    Adding time.sleep(5) makes it import correctly, there's some kind of race condition here.

    – F.X.
    Oct 22 '18 at 17:01















1















The following code tries to create-then-import two modules:



# coding: utf-8

import os
import time

# Remove the modules we're about to create if they already exist
def force_unlink(name):
try:
os.unlink(name)
except OSError:
pass
force_unlink("print1.py")
force_unlink("print1.pyc")
force_unlink("print2.py")
force_unlink("print2.pyc")
time.sleep(1)

# Create module 1 and module 2, then try to import them just afterwards
print("Creating module 1...")
with open("print1.py", "wb+") as fd:
fd.write(b'print("Imported module 1")')
import print1
print("Creating module 2...")
with open("print2.py", "wb+") as fd:
fd.write(b'print("Imported module 2")')
import print2


On Windows, both imports work under Python 2 (2.7), but not under Python 3 (3.5 and 3.6):



$ python2 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Imported module 2
$ python3 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Traceback (most recent call last):
File "reproduce.py", line 26, in <module>
import print2
ImportError: No module named 'print2'


Adding time.sleep(5) before each import printX call makes it work.



Why is that?



Note: This is a simpler version of an issue I'm trying to figure out.










share|improve this question
























  • The line numbers in your posted code don't match the line numbers in your error message. Can you run the version of the code you posted, and post the results of that?

    – user2357112
    Oct 22 '18 at 16:44











  • @user2357112 The original script had coding: utf-8 on top. I added that, but it shouldn't really affect the results.

    – F.X.
    Oct 22 '18 at 16:49







  • 1





    Works fine in 3.7 on windows a well.

    – tgikal
    Oct 22 '18 at 16:51











  • Try generating temporary directories to put the files in, then import those. Or try writing the file into a temporary location, then move it into the final location (renames could trigger a different codepath). Or simpler still: write both files, then import them both after writing. Then generate a third and try to import that, and see if that 3rd module now fails. If that confirms there is a cache, try to see where the cache is located, in Python or on Windows. Use os.listdir(). Use os.scandir(), etc.

    – Martijn Pieters
    Oct 22 '18 at 16:56






  • 1





    Adding time.sleep(5) makes it import correctly, there's some kind of race condition here.

    – F.X.
    Oct 22 '18 at 17:01













1












1








1








The following code tries to create-then-import two modules:



# coding: utf-8

import os
import time

# Remove the modules we're about to create if they already exist
def force_unlink(name):
try:
os.unlink(name)
except OSError:
pass
force_unlink("print1.py")
force_unlink("print1.pyc")
force_unlink("print2.py")
force_unlink("print2.pyc")
time.sleep(1)

# Create module 1 and module 2, then try to import them just afterwards
print("Creating module 1...")
with open("print1.py", "wb+") as fd:
fd.write(b'print("Imported module 1")')
import print1
print("Creating module 2...")
with open("print2.py", "wb+") as fd:
fd.write(b'print("Imported module 2")')
import print2


On Windows, both imports work under Python 2 (2.7), but not under Python 3 (3.5 and 3.6):



$ python2 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Imported module 2
$ python3 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Traceback (most recent call last):
File "reproduce.py", line 26, in <module>
import print2
ImportError: No module named 'print2'


Adding time.sleep(5) before each import printX call makes it work.



Why is that?



Note: This is a simpler version of an issue I'm trying to figure out.










share|improve this question
















The following code tries to create-then-import two modules:



# coding: utf-8

import os
import time

# Remove the modules we're about to create if they already exist
def force_unlink(name):
try:
os.unlink(name)
except OSError:
pass
force_unlink("print1.py")
force_unlink("print1.pyc")
force_unlink("print2.py")
force_unlink("print2.pyc")
time.sleep(1)

# Create module 1 and module 2, then try to import them just afterwards
print("Creating module 1...")
with open("print1.py", "wb+") as fd:
fd.write(b'print("Imported module 1")')
import print1
print("Creating module 2...")
with open("print2.py", "wb+") as fd:
fd.write(b'print("Imported module 2")')
import print2


On Windows, both imports work under Python 2 (2.7), but not under Python 3 (3.5 and 3.6):



$ python2 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Imported module 2
$ python3 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Traceback (most recent call last):
File "reproduce.py", line 26, in <module>
import print2
ImportError: No module named 'print2'


Adding time.sleep(5) before each import printX call makes it work.



Why is that?



Note: This is a simpler version of an issue I'm trying to figure out.







python windows python-import






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 15 '18 at 15:00







F.X.

















asked Oct 22 '18 at 16:30









F.X.F.X.

3,52333054




3,52333054












  • The line numbers in your posted code don't match the line numbers in your error message. Can you run the version of the code you posted, and post the results of that?

    – user2357112
    Oct 22 '18 at 16:44











  • @user2357112 The original script had coding: utf-8 on top. I added that, but it shouldn't really affect the results.

    – F.X.
    Oct 22 '18 at 16:49







  • 1





    Works fine in 3.7 on windows a well.

    – tgikal
    Oct 22 '18 at 16:51











  • Try generating temporary directories to put the files in, then import those. Or try writing the file into a temporary location, then move it into the final location (renames could trigger a different codepath). Or simpler still: write both files, then import them both after writing. Then generate a third and try to import that, and see if that 3rd module now fails. If that confirms there is a cache, try to see where the cache is located, in Python or on Windows. Use os.listdir(). Use os.scandir(), etc.

    – Martijn Pieters
    Oct 22 '18 at 16:56






  • 1





    Adding time.sleep(5) makes it import correctly, there's some kind of race condition here.

    – F.X.
    Oct 22 '18 at 17:01

















  • The line numbers in your posted code don't match the line numbers in your error message. Can you run the version of the code you posted, and post the results of that?

    – user2357112
    Oct 22 '18 at 16:44











  • @user2357112 The original script had coding: utf-8 on top. I added that, but it shouldn't really affect the results.

    – F.X.
    Oct 22 '18 at 16:49







  • 1





    Works fine in 3.7 on windows a well.

    – tgikal
    Oct 22 '18 at 16:51











  • Try generating temporary directories to put the files in, then import those. Or try writing the file into a temporary location, then move it into the final location (renames could trigger a different codepath). Or simpler still: write both files, then import them both after writing. Then generate a third and try to import that, and see if that 3rd module now fails. If that confirms there is a cache, try to see where the cache is located, in Python or on Windows. Use os.listdir(). Use os.scandir(), etc.

    – Martijn Pieters
    Oct 22 '18 at 16:56






  • 1





    Adding time.sleep(5) makes it import correctly, there's some kind of race condition here.

    – F.X.
    Oct 22 '18 at 17:01
















The line numbers in your posted code don't match the line numbers in your error message. Can you run the version of the code you posted, and post the results of that?

– user2357112
Oct 22 '18 at 16:44





The line numbers in your posted code don't match the line numbers in your error message. Can you run the version of the code you posted, and post the results of that?

– user2357112
Oct 22 '18 at 16:44













@user2357112 The original script had coding: utf-8 on top. I added that, but it shouldn't really affect the results.

– F.X.
Oct 22 '18 at 16:49






@user2357112 The original script had coding: utf-8 on top. I added that, but it shouldn't really affect the results.

– F.X.
Oct 22 '18 at 16:49





1




1





Works fine in 3.7 on windows a well.

– tgikal
Oct 22 '18 at 16:51





Works fine in 3.7 on windows a well.

– tgikal
Oct 22 '18 at 16:51













Try generating temporary directories to put the files in, then import those. Or try writing the file into a temporary location, then move it into the final location (renames could trigger a different codepath). Or simpler still: write both files, then import them both after writing. Then generate a third and try to import that, and see if that 3rd module now fails. If that confirms there is a cache, try to see where the cache is located, in Python or on Windows. Use os.listdir(). Use os.scandir(), etc.

– Martijn Pieters
Oct 22 '18 at 16:56





Try generating temporary directories to put the files in, then import those. Or try writing the file into a temporary location, then move it into the final location (renames could trigger a different codepath). Or simpler still: write both files, then import them both after writing. Then generate a third and try to import that, and see if that 3rd module now fails. If that confirms there is a cache, try to see where the cache is located, in Python or on Windows. Use os.listdir(). Use os.scandir(), etc.

– Martijn Pieters
Oct 22 '18 at 16:56




1




1





Adding time.sleep(5) makes it import correctly, there's some kind of race condition here.

– F.X.
Oct 22 '18 at 17:01





Adding time.sleep(5) makes it import correctly, there's some kind of race condition here.

– F.X.
Oct 22 '18 at 17:01












1 Answer
1






active

oldest

votes


















2














I think I know what is going on. The new Python 3 import machinery caches the filenames it finds in directories. It will reload the cache when the mtime, the modification time, of the directory changes.



See the importlib._bootstrap_external.FileFinder.find_spec() method implementation, which contains:



try:
mtime = _path_stat(self.path or _os.getcwd()).st_mtime
except OSError:
mtime = -1
if mtime != self._path_mtime:
self._fill_cache()
self._path_mtime = mtime


Here _path_stat is just a os.stat() call, but localised to avoid imports. The _fill_cache() method executes the os.listdir() call.



On some Windows filesystems, the resolution of mtime is notoriously low, up to 2 seconds. For your case, the resolution is apparently still low enough for the cache to not be updated by the time you try to load the second module. Although the NTFS filesystem can record times in 100ns increments, in practice the limiting factor there appears to be the Windows system clock, which I understand is usually limited to a resolution of 15ms. So if you write print2.py within 15ms of writing print1.py, then Python won't notice.



Python does give you the means to clear this cache; use the importlib.invalidate_caches() method; this will reset the _path_mtime attribute on the FileFinder instance back to -1, forcing a new _fill_cache() call.



As the function's documentation states:




This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.







share|improve this answer
























    Your Answer






    StackExchange.ifUsing("editor", function ()
    StackExchange.using("externalEditor", function ()
    StackExchange.using("snippets", function ()
    StackExchange.snippets.init();
    );
    );
    , "code-snippets");

    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "1"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f52933869%2fwhy-does-importing-fail-after-creating-a-module-under-python-3-on-windows%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    2














    I think I know what is going on. The new Python 3 import machinery caches the filenames it finds in directories. It will reload the cache when the mtime, the modification time, of the directory changes.



    See the importlib._bootstrap_external.FileFinder.find_spec() method implementation, which contains:



    try:
    mtime = _path_stat(self.path or _os.getcwd()).st_mtime
    except OSError:
    mtime = -1
    if mtime != self._path_mtime:
    self._fill_cache()
    self._path_mtime = mtime


    Here _path_stat is just a os.stat() call, but localised to avoid imports. The _fill_cache() method executes the os.listdir() call.



    On some Windows filesystems, the resolution of mtime is notoriously low, up to 2 seconds. For your case, the resolution is apparently still low enough for the cache to not be updated by the time you try to load the second module. Although the NTFS filesystem can record times in 100ns increments, in practice the limiting factor there appears to be the Windows system clock, which I understand is usually limited to a resolution of 15ms. So if you write print2.py within 15ms of writing print1.py, then Python won't notice.



    Python does give you the means to clear this cache; use the importlib.invalidate_caches() method; this will reset the _path_mtime attribute on the FileFinder instance back to -1, forcing a new _fill_cache() call.



    As the function's documentation states:




    This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.







    share|improve this answer





























      2














      I think I know what is going on. The new Python 3 import machinery caches the filenames it finds in directories. It will reload the cache when the mtime, the modification time, of the directory changes.



      See the importlib._bootstrap_external.FileFinder.find_spec() method implementation, which contains:



      try:
      mtime = _path_stat(self.path or _os.getcwd()).st_mtime
      except OSError:
      mtime = -1
      if mtime != self._path_mtime:
      self._fill_cache()
      self._path_mtime = mtime


      Here _path_stat is just a os.stat() call, but localised to avoid imports. The _fill_cache() method executes the os.listdir() call.



      On some Windows filesystems, the resolution of mtime is notoriously low, up to 2 seconds. For your case, the resolution is apparently still low enough for the cache to not be updated by the time you try to load the second module. Although the NTFS filesystem can record times in 100ns increments, in practice the limiting factor there appears to be the Windows system clock, which I understand is usually limited to a resolution of 15ms. So if you write print2.py within 15ms of writing print1.py, then Python won't notice.



      Python does give you the means to clear this cache; use the importlib.invalidate_caches() method; this will reset the _path_mtime attribute on the FileFinder instance back to -1, forcing a new _fill_cache() call.



      As the function's documentation states:




      This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.







      share|improve this answer



























        2












        2








        2







        I think I know what is going on. The new Python 3 import machinery caches the filenames it finds in directories. It will reload the cache when the mtime, the modification time, of the directory changes.



        See the importlib._bootstrap_external.FileFinder.find_spec() method implementation, which contains:



        try:
        mtime = _path_stat(self.path or _os.getcwd()).st_mtime
        except OSError:
        mtime = -1
        if mtime != self._path_mtime:
        self._fill_cache()
        self._path_mtime = mtime


        Here _path_stat is just a os.stat() call, but localised to avoid imports. The _fill_cache() method executes the os.listdir() call.



        On some Windows filesystems, the resolution of mtime is notoriously low, up to 2 seconds. For your case, the resolution is apparently still low enough for the cache to not be updated by the time you try to load the second module. Although the NTFS filesystem can record times in 100ns increments, in practice the limiting factor there appears to be the Windows system clock, which I understand is usually limited to a resolution of 15ms. So if you write print2.py within 15ms of writing print1.py, then Python won't notice.



        Python does give you the means to clear this cache; use the importlib.invalidate_caches() method; this will reset the _path_mtime attribute on the FileFinder instance back to -1, forcing a new _fill_cache() call.



        As the function's documentation states:




        This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.







        share|improve this answer















        I think I know what is going on. The new Python 3 import machinery caches the filenames it finds in directories. It will reload the cache when the mtime, the modification time, of the directory changes.



        See the importlib._bootstrap_external.FileFinder.find_spec() method implementation, which contains:



        try:
        mtime = _path_stat(self.path or _os.getcwd()).st_mtime
        except OSError:
        mtime = -1
        if mtime != self._path_mtime:
        self._fill_cache()
        self._path_mtime = mtime


        Here _path_stat is just a os.stat() call, but localised to avoid imports. The _fill_cache() method executes the os.listdir() call.



        On some Windows filesystems, the resolution of mtime is notoriously low, up to 2 seconds. For your case, the resolution is apparently still low enough for the cache to not be updated by the time you try to load the second module. Although the NTFS filesystem can record times in 100ns increments, in practice the limiting factor there appears to be the Windows system clock, which I understand is usually limited to a resolution of 15ms. So if you write print2.py within 15ms of writing print1.py, then Python won't notice.



        Python does give you the means to clear this cache; use the importlib.invalidate_caches() method; this will reset the _path_mtime attribute on the FileFinder instance back to -1, forcing a new _fill_cache() call.



        As the function's documentation states:




        This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.








        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Oct 22 '18 at 17:35

























        answered Oct 22 '18 at 17:27









        Martijn PietersMartijn Pieters

        715k13825002311




        715k13825002311





























            draft saved

            draft discarded
















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f52933869%2fwhy-does-importing-fail-after-creating-a-module-under-python-3-on-windows%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Top Tejano songwriter Luis Silva dead of heart attack at 64

            政党

            天津地下鉄3号線