WTS L2 SteamUpdater

[MajorSpear]

Cron job* sorry typo.

 

Well sorry; it's interesting tho, good luck ^^

[xxdem]

New feature

 

Package installations:

• Detects when its faster to use a specific package
• Download many files in a custom tarball compressed with gzip
• extracts only the needed data
• detects first time installation to download faster a single package

[xxdem]

Working on this again, also decided to share a beta version so you can see how insanely fast this thing is.

 

Test Server

 

 

CPU: VPS 2vCore

SSD

Bandwidth: 100 mbps

HTTP daemon: Lighttpd

OS: Archlinux

 

 

 

Test Client

 

 

CPU: i7 6950X

SSD 850 Pro

Bandwidth 50mbps

OS: Win10 x64

 

File Scan: < 20ms

Full update: < 25 seconds

 

 

 

Beta client can be downloaded bellow, it will try to download a file patch of size ~= 120mb.

The executable is not signed and windows or chrome may warn you not to download it, but there's nothing to worry about Virus Total

 

The bellow client may go into non-working state any time because the instructions offsets may get updated

 

Download Link

 

The message Initializing...l means that the updater done it's work and waits for L2 client to launch but this feature is not yet implemented, just hit the Cancel button

Edited April 7, 2017 by xxdem

[xxdem]

PS: ALL the files on this updater are automatically considered as "Critical updates" this updater in other words will force "Full Check" everytime its run. A "Full Check" scenario on an old-obsolete updater is about 5-8 minutes on an average patch size, this updater is able to use 100% of the available hardware resources and give's the ability to read/scan the same patch within 50-100 MS

Edited April 7, 2017 by xxdem

[xxdem]

Update

 

After lots of work, succeeded onto making stream decompression n-memory while downloading. This feature is totally unique among all updaters and I will explain why.

 

Technical explanation of how it works

 

Old mainstream method

 

 

Download all bytes and append/write intto xxx.zip on the hard disk

Buffered reading of xxx.zip and decompressing in memory again

Stream de-compression of in-memory bytes back to the harddisk and appending into un-compressed file xxx.u

 

A total of 3 steps involving heavy IO operations

 

 

 

In memory (new method)

 

 

Download next available data chunk -> decompress in-memory -> append to xxx.u

 

A total of 1 step reducing IO about 66% in comparison with the old method. It won't bottleneck your download speed since it operates in different worker thread.

 

 

[Stive]

do you have any skype?

[xxdem]

do you have any skype?

 

Not using skype anymore, please PM me here for start.

[Elfo]

a

Edited October 1, 2021 by Elfocrash

[xxdem]

Threading has nothing to do with bottlenecking the connection.

This connection bottleneck is caused because the download function is trying to use as much bandwidth as possible so it can be downloaded as fast as possible.

 

Isn't that correct?

 

In theory, if u put additional workload on the very same thread that downloads through a socket, you add latency on the connection eventually decreasing network efficiency

[Elfo]

a

Edited October 1, 2021 by Elfocrash

[pirama]

your virustotal need update 

virustotal

[xxdem]

Isn't there a better way to throttle the bandwidth of the download tho?

 

There is no throttle involved during a normal download, the old version or even the standard updater we are all used to will download the file with the highest efficiency possible. But will then require to decompress the compressed file into a new file on the disk with it's own new size, this means delete the downloaded compressed file and decompress it into a new file, which I believe adds unnecessary IO.

My method is more simple, it will decompress the bytes as soon as they are downloaded from the socket while downloading, this method will introduce input lag on the download thread and will throttle the network down, so this was solved with a second worker thread that received job from the download thread, decompressing the bytes in a streamed writer who is also responsible of writing the decompressed bytes to the hardisk through another buffered writer

Edited April 19, 2017 by xxdem

[xxdem]

your virustotal need update 

virustotal

 

yes its a virus, I want to scam your endless money

 

endgame antivirus

 

Edited April 19, 2017 by xxdem

[pirama]

yes its a virus, I want to scam your endless money

 

endgame antivirus

 

really you are so idiot ?

i said only you scan need update nothing more ...

i'm sure you don't have the Knowledge to create anything like this ...

Edited April 25, 2017 by pirama

[xxdem]

i'm sure you don't have the Knowledge to create anything like this ...

 

that charms me thank you