Move Files makes sometimes a file disappear

[NewsArchive]

Hi Friedrich,

I have the follopwing line in my script:

Move File(s) from "%DATENTOUPDATEDIR%\*.bdb" to "%WORKERDIR%"

It Moves about 700 Files and up to 30GB.

Sometimes (around 1 of 2.000.000) a random file out of these 700 files
is deleted in the source, but not created in the target.

I've checked, that the antivirus program does not have deleted the file.

I guess, you copy the file and delte the original one after that.

Is there any reason, why you don't move the file(s) instead of copy and
delete, if the source and the target resides on the same drive?

Is it possible, to do more error checks before deleting the source?

It's a rare symptom and I can't reproduce it, but it kills the data of
our customers and it generates huge costs (>10.000Eur till now)

Thanks in advance for your help.

Markus

[NewsArchive]

Hi Markus,

>
> Is it possible, to do more error checks before deleting the source?
>

The "Move File(s)" function uses the CopyFile Windows API to copy the file
from A to B. If CopyFile returns a nonzero value (= success) then (and only
then) the DeleteFile Windows API is executed and removes the source file.
If CopyFile returns zero (failed), the DeleteFile API is not executed and
the source is not deleted.

--
Friedrich Linder
Lindersoft
www.lindersoft.com
+1.954.252.3910

SetupBuilder is Windows 7 installation -- "point. click. ship"

-- Official Comodo Code Signing and SSL Certificate Partner

[NewsArchive]

BTW, I have attached a screenshot from the original code snipped that
"moves" the file from A to B.

Friedrich

[NewsArchive]

Hi Friedrich,

thanks for your answer. I believe what you wrote and I see it in your
screenshot, but, unfortunately, this doesn't help me.

Is it possible, that file is copied to the wrong destination?

Do you have an other explanation why the file is vanished or a
suggestion, what I can test?

And again: What is the reason, why you don't use
"MoveFileEx"?(http://msdn.microsoft.com/en-us/library/aa365240(v=VS.85).aspx)

What is the line "LogInfo" in your Screenshot? How can I activate this
logfile?

Thanks in advance
Markus

[NewsArchive]

Hi Markus,

> Do you have an other explanation why the file is vanished or a suggestion,
> what I can test?

Sorry, I do not have an explanation. But IMO, it's not related to
SetupBuilder (yeah, of course <g>)

If you have the uninstall .log enabled in your project and the "Remove on
Uninstall" option enabled in the Move File(s) function, you should have a
log entry "043" with the name of the "moved" file in your uninstall .log.

Something like:
043|C:\foldertest\test\movefile.gif

We can't use the MoveFileEx API because there are still quite a few users
(~20% of the SetupBuilder user base) supporting Win9x systems. There is
nothing wrong with using a combination of CopyFile/DeleteFile here. Of
course, we would see a (huge) speed improvement when using MoveFile, but it
can't be used when copying to a different volume. Dynamically loading
MoveFileEx on systems that support it would also be possible, but I don't
see any problem with our code as it is now. As you can see from the source
code that I posted, DeleteFile() is only executed if CopyFile() succeeded.
So Windows reports that the file has been copied successfully and *after*
that, the installer removes the original file. I can't think of a scenario
where Windows CopyFile incorrectly returns "success".

By the way, the very same values for "FileToCopy" and "FileToCopyTo" would
be used in a MoveFile(Ex) API scenario.

Friedrich

[NewsArchive]

Thanks friedrich,

I will give the uninstall log a try.

is it possible, to have the uninstall log without an uninstall.exe? The
problem is caused by a tool build with SB, not an installer :-)

I would love to see MoveFileEx dynamically load (perhaps in SB 7.3?), if
possible. The speed improvement is it worth.

Markus

[NewsArchive]

Hi Markus,

> I will give the uninstall log a try.
>
> is it possible, to have the uninstall log without an uninstall.exe? The
> problem is caused by a tool build with SB, not an installer :-)

Aha, okay ;-)

The following "should" work:

1. In "General Information", set "Support Uninstall" to "Yes"

2. Clear the "Uninstall Name [UNINSTALL]" field

3. The "Uninstall Log [UNINSTALL_LOG]" should be:
%_SB_INSTALLDIR%\Uninst_[PRODUCTNAME].log

4. Your script should have the following line:
Create Installation App & Log "[UNINSTALL_LOG]"

Now make sure that %_SB_INSTALLDIR% points to a folder that exists and the
application should create a "uninstall" .log (but no .exe).

Also set "Use Add/Remove Programs" in the General Properties to "No"

> I would love to see MoveFileEx dynamically load (perhaps in SB 7.3?), if
> possible. The speed improvement is it worth.

Yes, it should be possible to add such an option. No problem.

Friedrich

[NewsArchive]

Thanks.

Markus Zander

[NewsArchive]

BTW, even MoveFileEx makes (internally) use of the CopyFile/DeleteFile API
combination when moving to a different volume:

MOVEFILE_COPY_ALLOWED = 'If the file is to be moved to a different volume,
the function simulates the move by using the CopyFile and DeleteFile
functions'

So the only difference is, that the SetupBuilder installer runtime (to be on
the safe side) always uses this CopyFile/DeleteFile method.

Friedrich

[NewsArchive]

But as you already mentioned, if source and destination resides on the
same drive, it's a big difference.

Markus

[NewsArchive]

> But as you already mentioned, if source and destination resides on the
> same drive, it's a big difference.

Yes, I agree 100%.

Friedrich

[NewsArchive]

Hi Friedrich,

I use my own movefile-sbi-script now, which has extensive logging and
error checking. For that, I can't test the problem anymore.

Markus

[NewsArchive]

Hi Markus,

> I use my own movefile-sbi-script now, which has extensive logging and
> error checking. For that, I can't test the problem anymore.

Okay, thanks for the info. We have added a note to the system. If this is
reported again, we'll reopen the ticket.

BTW, we tested this on several machines and "moved" 1,000 files (~35 GB).
It never failed.

Friedrich

[NewsArchive]

Hi Friedrich,

I still have a strange behaviour with "move files" (SB 7.4.3313):

First, have a look at these script snippet:

==== BEGINN SNIPPET ====

Set Variable %[L]LOOPCOUNT% to "0"
Install Event Entry: "Beginne MoveFiles %[L]SOURCE% ==> %[L]TARGET%
(Overwrite = %[L]OVERWRITE%)"
Loop
Set Variable %[L]LOOPCOUNT% to (Increment by 1)
If %[L]OVERWRITE% Equals "$TRUE$" Then
Move File(s) from "%[L]SOURCE%" to "%[L]TARGET%"
Else
Move File(s) from "%[L]SOURCE%" to "%[L]TARGET%"
End
If %_SB_RETURNEX% Equals "0" Then
Break Loop
Else
If %_SB_ERRORCODE% Equals "0" Then
Install Event Entry: "Beim Verschieben der Dateien ist ein
Problem aufgetreten."
Display Message Box "Beim Verschieben von Dateien ist ein
Problem aufge..." -- "Problem beim Verschieben von Dateien"
If %_SB_RETURN% Equals "$IDABORT$" Then
Exit Installation(1)
ElseIf %_SB_RETURN% Equals "$IDIGNORE$" Then
Install Event Entry: "Nutzer möchte den Fehler ignorieren."
Set Variable %_SB_PARAM1% to "2"
Else
Install Event Entry: "Nutzer möchte die letzte Aktion
wiederholen."
Cycle Loop
End
End
If %[L]LOOPCOUNT% Greater Than "1" Then
Display Message Box "Beim Verschieben von Dateien ist ein
ungewöhnliche..." -- "Ungewöhnliches Ereignis"
Install Event Entry: "Es waren %[L]LOOPCOUNT% Durchläufe zum
Verschieben der Dateien erforderlich!"
End
End
End
Install Event Entry: "Beende MoveFiles"

==== END SNIPPET ====



And here is the corresponding sbevents.txt snippet:

==== BEGINN SNIPPET ====

|05/13/2011|15:11:46:334|Beginne MoveFiles C:\Daten\AIDA6\Daten\*.bdb
==> C:\Daten\AIDA6\Daten\W (Overwrite = C:\Daten\AIDA6\Daten\W)
|05/13/2011|15:11:46:334|@SETVARIABLE (0:1:2)
|05/13/2011|15:11:46:334|Set Variable %MOVEFILESLOOPCOUNT% to 1
|05/13/2011|15:12:14:867|Set Variable %_SB_RETURNEX% to 681
|05/13/2011|15:12:14:867|@SETVARIABLE (0:1:2)
|05/13/2011|15:12:14:867|Set Variable %MOVEFILESLOOPCOUNT% to 2
|05/13/2011|15:12:17:238|Set Variable %_SB_RETURNEX% to 19
|05/13/2011|15:12:17:238|@DISPLAYMESSAGE Beim Verschieben von Dateien
ist ein ungewöhnliches Ereignis aufgetreten, \nDBUpdater konnte die
Situation jedoch klären.\n\nBitte Senden Sie nach dem Update das Logfile
an AIDA. Sie finden das Logfile direkt auf C:\. : Ungewöhnliches Ereignis
|05/13/2011|15:14:27:120|Set Variable %_SB_RETURN% to 1
|05/13/2011|15:14:27:120|Es waren 2 Durchläufe zum Verschieben der
Dateien erforderlich!
|05/13/2011|15:14:27:120|@SETVARIABLE (0:1:2)
|05/13/2011|15:14:27:120|Set Variable %MOVEFILESLOOPCOUNT% to 3
|05/13/2011|15:14:27:120|Set Variable %_SB_RETURNEX% to 0
|05/13/2011|15:14:27:136|Beende MoveFiles

==== END SNIPPET ====

You can cleary see, that SB has needed two attempts to move all files. I
can't reproduce this problem at will, but since I've added these Message
Box, I have reports from our customers from time to time.

I would be really nice, if SB says x files moved (in %_SB_RETURNEX%) and
no errors occured (%_SB_ERRORCODE != 0), that this is true, but I can't
trust and need the upper include script, which is not more or less than
an ugly workaround.

Please, could you see, what you can do to fix this issue?

Thanks in advance.
Markus

[NewsArchive]

Hi Markus,

> You can cleary see, that SB has needed two attempts to move all files. I
> can't reproduce this problem at will, but since I've added these Message
> Box, I have reports from our customers from time to time.
>
> I would be really nice, if SB says x files moved (in %_SB_RETURNEX%) and
> no errors occured (%_SB_ERRORCODE != 0), that this is true, but I can't
> trust and need the upper include script, which is not more or less than an
> ugly workaround.
>
> Please, could you see, what you can do to fix this issue?

The "MoveFile(s)..." script function is very simple. It loops through all
files in a folder tree and then uses the CopyFile Windows API to copy it
from A to B. If this succeeded, it tries to DeleteFile A.

BTW, it's possible that the error flag of a failed CopyFile process is
overwritten by the next successful CopyFile action and so %_SB_ERRORCODE%
might be incorrect. We already fixed this in the latest internal source
code.

And we have added an option to explicitly use the MoveFile Windows API
(instead a combination of CopyFile/DeleteFile) in the "MoveFile(s)..."
script function. If you move files on a different volume, you can enable
this option.

Friedrich

[NewsArchive]

(sync from SB8)

Friedrich

[NewsArchive]

IMPROVEMENT: (sync from SB8) [SB#105135] Add "Use MoveFile API" option to
the "Move File(s)..." script function.

FIX : (sync from SB8) [SB#105133] Runtime: Under certain
circumstances, the "Copy File(s)..." script function returned
an incorrect %_SB_ERRORCODE% value.

FIX : (sync from SB8) [SB#105134] Runtime: Under certain
circumstances, the "Move File(s)..." script function returned
an incorrect %_SB_ERRORCODE% value.

--
Friedrich Linder
Lindersoft
www.lindersoft.com
+1.954.252.3910

SetupBuilder is Windows 7 installation -- "point. click. ship"

-- Official Comodo Code Signing and SSL Certificate Partner

[NewsArchive]

Hi Friedrich,

could you please send me a development build? I have the next report
concerning this matter.

Thank you.
Markus

[NewsArchive]

Markus,

> could you please send me a development build? I have the next report
> concerning this matter.

We do have internal tests running. We'll try to make a development build
available within the next 24 hours.

Friedrich

[NewsArchive]

Hi Friedrich,

after you had added the option to use the Windows Move File API, there
was no problem with this til now...

Last week, I updatet from SB7 to SB8 and now, I receive a logfile, which
shows, that not all files were moved.


Logfile:
..
..
..
|03/27/2013|12:05:45.719|sbkernel: 8.0.4000 : WinID: 1024 HTTP_Timeout : 0
..
..
..
|03/27/2013|12:21:12.111|Beginne MoveFiles mit SB>= 74_3325:
K:\PWS\AIDA\DATEN_~1.13\W\*.* ==> K:\PWS\AIDA\DATEN_~1.13 (Overwrite = 1)
|03/27/2013|12:21:12.111|@SETVARIABLE (0 : 1 : 2)
|03/27/2013|12:21:12.111|Set Variable %MOVEFILESLOOPCOUNT% to 1
|03/27/2013|12:21:41.798|Set Variable %_SB_RETURNEX% to 611
|03/27/2013|12:21:41.798|@SETVARIABLE (0 : 1 : 2)
|03/27/2013|12:21:41.798|Set Variable %MOVEFILESLOOPCOUNT% to 2
|03/27/2013|12:21:48.100|Set Variable %_SB_RETURNEX% to 167
|03/27/2013|12:21:48.100|@DISPLAYMESSAGE Beim Verschieben von Dateien
ist ein ungewöhnliches Ereignis aufgetreten, \nDBUpdater konnte die
Situation jedoch klären.\n\nBitte Senden Sie nach dem Update das Logfile
an AIDA. Sie finden das Logfile direkt auf C:\. : Ungewöhnliches Ereignis


Script:
!
//////////////////////////////////////////////////////////////////////////////////////////////////
! //
! // Verschiebt Dateien auf sichere Art und Weise von einem Verzeichnis
in ein anderes
! //
!
//////////////////////////////////////////////////////////////////////////////////////////////////

#set compiler variable [MOVEFILES_SOURCE] = "" [Config]
Set Variable %VAL1% to "[MOVEFILES_SOURCE]"

#set compiler variable [MOVEFILES_TARGET] = "" [Config]
Set Variable %VAL2% to "[MOVEFILES_TARGET]"

#set compiler variable [MOVEFILES_OVERWRITE] = "$FALSE$" [Config]
Set Variable %VAL3% to "[MOVEFILES_OVERWRITE]"

#include script "namespacestart.sbi" from "[_PATH_INCLUDESCRIPTS]"
[Config] ==> [LC]=MOVEFILES

Set Variable %[L]SOURCE% to "%VAL1%"
Set Variable %[L]TARGET% to "%VAL2%"
Set Variable %[L]OVERWRITE% to "%VAL3%"

Set Variable %[L]LOOPCOUNT% to "0"
Install Event Entry: "Beginne MoveFiles mit SB>= 74_3325: %[L]SOURCE%
==> %[L]TARGET% (Overwrite = %[L]OVERWRITE%)"
Loop
Set Variable %[L]LOOPCOUNT% to (Increment by 1)
If %[L]OVERWRITE% Equals "$TRUE$" Then
Move File(s) from "%[L]SOURCE%" to "%[L]TARGET%" [Permanent]
[MoveFileAPI]
Else
Move File(s) from "%[L]SOURCE%" to "%[L]TARGET%" [Skip]
[Permanent] [MoveFileAPI]
End
If %_SB_RETURNEX% Equals "0" Then
Break Loop
Else
If %_SB_ERRORCODE% Equals "0" Then
Install Event Entry: "Beim Verschieben der Dateien ist mit
SB>=7_4_3325 ein Problem aufgetreten."
Display Message Box "Beim Verschieben von Dateien ist ein
Problem aufge..." -- "Problem beim Verschieben von Dateien"
If %_SB_RETURN% Equals "$IDABORT$" Then
Exit Installation(1)
ElseIf %_SB_RETURN% Equals "$IDIGNORE$" Then
Install Event Entry: "Nutzer möchte den Fehler ignorieren."
Set Variable %_SB_PARAM1% to "2"
Else
Install Event Entry: "Nutzer möchte die letzte Aktion
wiederholen."
Cycle Loop
End
End
If %[L]LOOPCOUNT% Greater Than "1" Then
Display Message Box "Beim Verschieben von Dateien ist ein
ungewöhnliche..." -- "Ungewöhnliches Ereignis"
Install Event Entry: "Es waren %[L]LOOPCOUNT% Durchläufe zum
Verschieben der Dateien erforderlich!"
End
End
End
Install Event Entry: "Beende MoveFiles"

#include script "namespaceend.sbi" from "[_PATH_INCLUDESCRIPTS]"


Any suggestions?

Thanks for your answer.

Markus

[NewsArchive]

Hi Markus,

NOTHING changed in this function from SB77 to SB8!

Friedrich

[NewsArchive]

Hi Friedrich,

thank you for you answer. So there are three points:

a) Have double checked, that SB77 and SB80 have identical sources for
this function? Have you checked, that the Windows Move file API really
is called?

b) What could cause this problem?

c) Have you considered, to "safe" your script function by a workaround?
You could do the loop and double check, which you can see in my include
script "under the hood".

Markus.

[NewsArchive]

Hi Markus,

> thank you for you answer. So there are three points:
>
> a) Have double checked, that SB77 and SB80 have identical
> sources for this function? Have you checked, that the Windows
> Move file API really is called?

Yes. I can confirm that not a single bit in that area changed from SB77 to
SB80. Last modification in this function is dated May 14, 2011 (SB75).

>
> b) What could cause this problem?
>

Any scenario that causes a MoveFile Windows API to fail. For example, an
over-protective or buggy anti-virus or anti-spyware system (false-positive),
a "locked" file scenario, etc.

> c) Have you considered, to "safe" your script function by a workaround?
> You could do the loop and double check, which you can see in my include
> script "under the hood".

There is no modification planned in this area. The function is very simple.
It LOOPS through a directory to build a list (QUEUE) of all files. After
that, it LOOPS through the QUEUE and calls the MoveFile Windows API to move
the file(s). If the MoveFile function fails, the processed file counter
gets not increased. The function does not trigger a "retry".

Friedrich

[NewsArchive]

Hi Friedrich,

OK, so I will rely on my include script. But to gather further
informations, could you please improve logging:

- How many Files do you have added to the "Queue"?
- Which files are failed?

Markus

[NewsArchive]

Markus,

> OK, so I will rely on my include script. But to gather further
> informations, could you please improve logging:
>
> - How many Files do you have added to the "Queue"?
> - Which files are failed?

Yes, no problem. I think we can add failed MoveFile actions (including the
"GetLastError" code) and the number of QUEUED/MOVED files to the event log.

Event logging will be further improved in the future.

Friedrich

[NewsArchive]

Thank you!

Markus Zander