Project

General

Profile

« Previous | Next » 

Revision 0c2e61fd

Added by David Sorber over 9 years ago

Adding an optimized version of the packJPG source to the source tree for this utility since it is entirely dependent on packJPG. I started to modify the Makefile but really I shoud just use CMake. I'll work on that next.

View differences:

software/photo_compress_archiver/Makefile
CXX=clang++
#~ CXX=g++
CXXFLAGS=-std=c++11 -g
OBJECTS = main.o PhotoCompressArchiver.o
LDFLAGS=-L/opt/local/lib -lboost_system-mt -lboost_filesystem-mt -lboost_program_options-mt -lboost_regex-mt
NUM_CORES:=$(shell grep -c ^processor /proc/cpuinfo)
PACKJPG_SRC = $(wildcard packJPG/*.cpp packJPG/*.h)
all: main
main: $(OBJECTS)
packjpg: $(PACKJPG_SRC)
@echo $(PACKJPG_SRC)
cd packJPG/build; cmake ..; make -j $(NUM_CORES)
cp packJPG/build/packjpg .
main: $(OBJECTS) packjpg
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o pca $(OBJECTS)
%.o : %.cc
$(CXX) $(CXXFLAGS) -I/opt/local/include -c $<
clean_packjpg:
rm -f packjpg
rm -rf packJPG/build/*
clean:
rm -f *.o
rm -f pca
software/photo_compress_archiver/packJPG/CMakeLists.txt
project("packJPG_opt")
cmake_minimum_required(VERSION 2.8)
include_directories(.)
set(packjpg_sources
../aricoder.cpp
../bitops.cpp
)
add_definitions("-std=c++1y -O3 -Wall -pedantic -DUNIX")
add_definitions("-funroll-loops -ffast-math -fsched-spec-load -fomit-frame-pointer")
#~ add_definitions("-static -static-libgcc -static-libstdc++")
add_executable(packjpg
${packjpg_sources}
../packjpg.cpp)
software/photo_compress_archiver/packJPG/Readme.txt
===============================================================================
DBS - 20170218
NOTE: This is a branch of the original packJPG code base that contains some
improvements. It came from here:
https://github.com/TarVanimelde/packJPG
At time of writing these improvements have been submitted as a pull request,
but it has not (yet) been accepted. The associated description is:
"Bitops and Aricoder Performance Enhancements, Visual Studio Compilation Fix"
I have remove unnecssary files (icons, .spec file, etc.) and added a
CMakeLists.txt CMake configuration file in favor of the existing Makefile.
===============================================================================
packJPG v2.5k (01/22/2016)
~~~~~~~~~~~~~~~~~~~~~~~~~~
packJPG is a compression program specially designed for further
compression of JPEG images without causing any further loss. Typically
it reduces the file size of a JPEG file by 20%.
LGPL v3 license and special permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All programs in this package are free software; you can redistribute
them and/or modify them under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either version 3
of the License, or (at your option) any later version.
The package is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
General Public License for more details at
http://www.gnu.org/copyleft/lgpl.html.
If the LGPL v3 license is not compatible with your software project you
might contact us and ask for a special permission to use the packJPG
library under different conditions. In any case, usage of the packJPG
algorithm under the LGPL v3 or above is highly advised and special
permissions will only be given where necessary on a case by case basis.
This offer is aimed mainly at closed source freeware developers seeking
to add PJG support to their software projects.
Copyright 2006...2014 by HTW Aalen University and Matthias Stirner.
Usage of packJPG
~~~~~~~~~~~~~~~~
JPEG files are compressed and PJG files are decompressed using this
command:
"packJPG [file(s)]"
packJPG recognizes file types on its own and decides whether to compress
(JPG) or decompress (PJG). For unrecognized file types no action is
taken. Files are recognized by content, not by extension.
packJPG supports wildcards like "*.*" and drag and drop of multiple
files. Filenames for output files are created automatically. In default
mode, files are never overwritten. If a filename is already in use,
packJPG creates a new filename by adding underscores.
If "-" is used as a filename input from stdin is assumed and output is
written to stdout. This can be useful for example if jpegtran is to be
used as a preprocessor.
Usage examples:
"packJPG *.pjg"
"packJPG lena.jpg"
"packJPG kodim??.jpg"
"packJPG - < sail.pjg > sail.jpg"
Command line switches
~~~~~~~~~~~~~~~~~~~~~
-ver verify files after processing
-v? level of verbosity; 0,1 or 2 is allowed (default 0)
-np no pause after processing files
-o overwrite existing files
-p proceed on warnings
-d discard meta-info
By default, compression is cancelled on warnings. If warnings are
skipped by using "-p", most files with warnings can also be compressed,
but JPEG files reconstructed from PJG files might not be bitwise
identical with the original JPEG files. There won't be any loss to
image data or quality however.
Unnecessary meta information can be discarded using "-d". This reduces
compressed files' sizes. Be warned though, reconstructed files won't be
bitwise identical with the original files and meta information will be
lost forever. As with "-p" there won't be any loss to image data or
quality.
There is no known case in which a file compressed by packJPG (without
the "-p" option, see above) couldn't be reconstructed to exactly the
state it was before. If you want an additional layer of safety you can
also use the verify option "-ver". In this mode, files are compressed,
then decompressed and the decompressed file compared to the original
file. If this test doesn't pass there will be an error message and the
compressed file won't be written to the drive.
Please note that the "-ver" option should never be used in conjunction
with the "-d" and/or "-p" options. As stated above, the "-p" and "-d"
options will most likely lead to reconstructed JPG files not being
bitwise identical to the original JPG files. In turn, the verification
process may fail on various files although nothing actually went wrong.
Usage examples:
"packJPG -v1 -o baboon.pjg"
"packJPG -ver lena.jpg"
"packJPG -d tiffany.jpg"
"packJPG -p *.jpg"
Known Limitations
~~~~~~~~~~~~~~~~~
packJPG is a compression program specially for JPEG files, so it doesn't
compress other file types.
packJPG has low error tolerance. JPEG files might not work with packJPG
even if they work perfectly with other image processing software. The
command line switch "-p" can be used to increase error tolerance and
compatibility.
If you try to drag and drop to many files at once, there might be a
windowed error message about missing privileges. In that case you can
try it again with less files or consider using the command prompt.
packJPG has been tested to work perfectly with thousands of files from
the command line. This issue also happens with drag and drop in other
applications, so it might not be a limitation of packJPG but a
limitation of Windows.
Compressed PJG files are not compatible between different packJPG
versions. You will get an error message if you try to decompress PJG
files with a different version than the one used for compression. You
may download older versions of packJPG from:
http://www.elektronik.htw-aalen.de/packJPG/binaries/old/
Open source release / developer info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The packJPG source codes is found inside the "source" subdirectory.
Additional documents aimed to developers, containing detailed
instructions on compiling the source code and using special
functionality, are included in the "packJPG" subdirectory.
History
~~~~~~~
v1.9a (04/20/2007) (non public)
- first released version
- only for testing purposes
v2.0 (05/28/2007) (public)
- first public version of packJPG
- minor improvements to overall compression
- minor bugfixes
v2.2 (08/05/2007) (public)
- around 40% faster compression & decompression
- major improvements to overall compression (around 2% on average)
- reading from stdin, writing to stdout
- smaller executable
- minor bugfixes
- various minor improvements
v2.3 (09/18/2007) (public)
- compatibility with JPEG progressive mode
- compatibility with JPEG extended sequential mode
- compatibility with the CMYK color space
- compatibility with older CPUs
- around 15% faster compression & decompression
- new switch: [-d] (discard meta-info)
- various bugfixes
v2.3a (11/21/2007) (public)
- crash issue with certain images fixed
- compatibility with packJPG v2.3 maintained
v2.3b (12/20/2007) (public)
- some minor errors in the packJPG library fixed
- compatibility with packJPG v2.3 maintained
v2.4 (03/24/2010) (public)
- major improvements (1%...2%) to overall compression
- around 10% faster compression & decompression
- major improvements to JPG compatibility
- size of executable reduced to ~33%
- new switch: [-ver] (verify file after processing)
- new switch: [-np] (no pause after processing)
- new progress bar output mode
- arithmetic coding routines rewritten from scratch
- various smaller improvements to numerous to list here
- new SFX (self extracting) archive format
v2.5 (11/11/2011) (public)
- improvements (~0.5%) to overall compression
- several minor bugfixes
- major code cleanup
- removed packJPX from the package
- added packARC to the package
- packJPG is now open source!
v2.5a (11/21/11) (public)
- source code compatibility improvements (Gerhard Seelmann)
- avoid some compiler warnings (Gerhard Seelmann)
- source code clean up (Gerhard Seelmann)
v2.5b (01/27/12) (public)
- further removal of redundant code
- some fixes for the packJPG static library
- compiler fix for Mac OS (thanks to Sergio Lopez)
- improved compression ratio calculation
- eliminated the need for temp files
v2.5c (04/13/12) (public)
- various source code optimizations
v2.5d (07/03/12) (public)
- fixed a rare bug with progressive JPEG
v2.5e (07/03/12) (public)
- some minor source code optimizations
- changed packJPG licensing to LGPL
- moved packARC to a separate package
v2.5f (02/24/13) (public)
- fixed a minor bug in the JPG parser (thanks to Stephan Busch)
v2.5g (09/14/13) (public)
- fixed a rare crash bug with manipulated JPEG files
v2.5h (12/07/13) (public)
- added a warning for inefficient huffman coding (thanks to Moinak Ghosh)
v2.5i (12/26/13) (public)
- fixed possible crash with malformed JPEG (thanks to Moinak Ghosh)
v2.5j (01/15/14) (public)
- various source code optimizations (using cppcheck)
v2.5k (01/22/16) (public)
- Updated contact info
- fixed a minor bug
Acknowledgements
~~~~~~~~~~~~~~~~
packJPG is the result of countless hours of research and development. It
is part of my final year project for Hochschule Aalen.
Prof. Dr. Gerhard Seelmann from Hochschule Aalen supported my
development of packJPG with his extensive knowledge in the field of data
compression. Without his advice, packJPG would not be possible.
The official developer blog for packJPG is hosted by encode.ru.
packJPG logo and icon are designed by Michael Kaufmann.
Contact
~~~~~~~
The official developer blog for packJPG:
http://packjpg.encode.ru/
For questions and bug reports:
packjpg (at) matthiasstirner.com
____________________________________
packJPG by Matthias Stirner, 01/2016
software/photo_compress_archiver/packJPG/aricoder.cpp
#include "aricoder.h"
#include "bitops.h"
#include <algorithm>
#include <functional>
#include <stdlib.h>
/* -----------------------------------------------
constructor for aricoder class
----------------------------------------------- */
aricoder::aricoder( iostream* stream, StreamMode iomode ) : sptr(stream), mode(iomode)
{
if ( mode == StreamMode::kRead) { // mode is reading / decoding
// code buffer has to be filled before starting decoding
for (uint32_t i = 0; i < CODER_USE_BITS; i++ )
ccode = ( ccode << 1 ) | read_bit();
} // mode is writing / encoding otherwise
}
/* -----------------------------------------------
destructor for aricoder class
----------------------------------------------- */
aricoder::~aricoder()
{
if ( mode == StreamMode::kWrite) { // mode is writing / encoding
// due to clow < CODER_LIMIT050, and chigh >= CODER_LIMIT050
// there are only two possible cases
if ( clow < CODER_LIMIT025 ) { // case a.)
write_bit<0>();
// write remaining bits
write_bit<1>();
writeNrbitsAsOne();
}
else { // case b.), clow >= CODER_LIMIT025
write_bit<1>();
} // done, zeroes are auto-read by the decoder
// pad code with zeroes
while (cbit > 0) {
write_bit<0>();
}
}
}
/* -----------------------------------------------
arithmetic encoder function
----------------------------------------------- */
void aricoder::encode( symbol* s )
{
// Make local copies of clow_ and chigh_ for cache performance:
uint32_t clow_local = clow;
uint32_t chigh_local = chigh;
// update steps, low count, high count
cstep = (chigh_local - clow_local + 1) / s->scale;
chigh_local = clow_local + (cstep * s->high_count) - 1;
clow_local = clow_local + (cstep * s->low_count);
// e3 scaling is performed for speed and to avoid underflows
// if both, low and high are either in the lower half or in the higher half
// one bit can be safely shifted out
while ( clow_local >= CODER_LIMIT050 || chigh_local < CODER_LIMIT050 ) {
if (chigh_local < CODER_LIMIT050 ) { // this means both, high and low are below, and 0 can be safely shifted out
// write 0 bit
write_bit<0>();
// shift out remaing e3 bits
writeNrbitsAsOne();
}
else { // if the first wasn't the case, it's clow >= CODER_LIMIT050
// write 1 bit
write_bit<1>();
clow_local &= CODER_LIMIT050 - 1;
chigh_local &= CODER_LIMIT050 - 1;
// shift out remaing e3 bits
writeNrbitsAsZero();
}
clow_local <<= 1;
chigh_local <<= 1;
chigh_local++;
}
// e3 scaling, to make sure that theres enough space between low and high
while ( (clow_local >= CODER_LIMIT025 ) && (chigh_local < CODER_LIMIT075 ) ) {
nrbits++;
clow_local &= CODER_LIMIT025 - 1;
chigh_local ^= CODER_LIMIT025 + CODER_LIMIT050;
// clow -= CODER_LIMIT025;
// chigh -= CODER_LIMIT025;
clow_local <<= 1;
chigh_local <<= 1;
chigh_local++;
}
clow = clow_local;
chigh = chigh_local;
}
void aricoder::writeNrbitsAsZero() {
if (nrbits + cbit >= 8) {
int remainingBits = 8 - cbit;
nrbits -= remainingBits;
bbyte <<= remainingBits;
sptr->write_byte(bbyte);
cbit = 0;
}
constexpr uint8_t zero = 0;
while (nrbits >= 8) {
sptr->write_byte(zero);
nrbits -= 8;
}
/*
No need to check if cbits is 8, since nrbits is strictly less than 8
and cbit is initially 0 here:
*/
bbyte <<= nrbits;
cbit += nrbits;
nrbits = 0;
}
void aricoder::writeNrbitsAsOne() {
if (nrbits + cbit >= 8) {
int remainingBits = 8 - cbit;
nrbits -= remainingBits;
bbyte <<= remainingBits;
bbyte |= std::numeric_limits<uint8_t>::max() >> (8 - remainingBits);
sptr->write_byte(bbyte);
cbit = 0;
}
constexpr uint8_t all_ones = std::numeric_limits<uint8_t>::max();
while (nrbits >= 8) {
sptr->write_byte(all_ones);
nrbits -= 8;
}
/*
No need to check if cbits is 8, since nrbits is strictly less than 8
and cbit is initially 0 here:
*/
bbyte = (bbyte << nrbits) | (std::numeric_limits<uint8_t>::max() >> (8 - nrbits));
cbit += nrbits;
nrbits = 0;
}
/* -----------------------------------------------
arithmetic decoder get count function
----------------------------------------------- */
unsigned int aricoder::decode_count( symbol* s )
{
// update cstep, which is needed to remove the symbol from the stream later
cstep = ( ( chigh - clow ) + 1 ) / s->scale;
// return counts, needed to decode the symbol from the statistical model
return ( ccode - clow ) / cstep;
}
/* -----------------------------------------------
arithmetic decoder function
----------------------------------------------- */
void aricoder::decode( symbol* s )
{
// no actual decoding takes place, as this has to happen in the statistical model
// the symbol has to be removed from the stream, though
// alread have steps updated from decoder_count
// update low count and high count
uint32_t ccode_local = ccode;
uint32_t clow_local = clow;
uint32_t chigh_local = clow_local + (cstep * s->high_count) - 1;
clow_local = clow_local + (cstep * s->low_count);
// e3 scaling is performed for speed and to avoid underflows
// if both, low and high are either in the lower half or in the higher half
// one bit can be safely shifted out
while ( (clow_local >= CODER_LIMIT050 ) || (chigh_local < CODER_LIMIT050 ) ) {
if (clow_local >= CODER_LIMIT050 ) {
clow_local &= CODER_LIMIT050 - 1;
chigh_local &= CODER_LIMIT050 - 1;
ccode_local &= CODER_LIMIT050 - 1;
} // if the first wasn't the case, it's chigh < CODER_LIMIT050
clow_local <<= 1;
chigh_local <<= 1;
chigh_local++;
ccode_local <<= 1;
ccode_local |= read_bit();
nrbits = 0;
}
// e3 scaling, to make sure that theres enough space between low and high
while ( (clow_local >= CODER_LIMIT025 ) && (chigh_local < CODER_LIMIT075 ) ) {
nrbits++;
clow_local &= CODER_LIMIT025 - 1;
chigh_local ^= CODER_LIMIT025 + CODER_LIMIT050;
// clow -= CODER_LIMIT025;
// chigh -= CODER_LIMIT025;
ccode_local -= CODER_LIMIT025;
clow_local <<= 1;
chigh_local <<= 1;
chigh_local++;
ccode_local <<= 1;
ccode_local |= read_bit();
}
chigh = chigh_local;
clow = clow_local;
ccode = ccode_local;
}
/* -----------------------------------------------
bit reader function
----------------------------------------------- */
unsigned char aricoder::read_bit()
{
// read in new byte if needed
if ( cbit == 0 ) {
if ( !sptr->read_byte(&bbyte)) // read next byte if available
bbyte = 0; // if no more data is left in the stream
cbit = 8;
}
// decrement current bit position
cbit--;
// return bit at cbit position
return BITN( bbyte, cbit );
}
/* -----------------------------------------------
universal statistical model for arithmetic coding
boundaries of this model:
max_s (maximum symbol) -> 1 <= max_s <= 1024 (???)
max_c (maximum context) -> 1 <= max_c <= 1024 (???)
max_o (maximum order) -> -1 <= max_o <= 4
c_lim (maximum count) -> 2 <= c_lim <= 4096 (???)
WARNING: this can be memory intensive, so don't overdo it
max_s == 256; max_c == 256; max_o == 4 would be way too much
----------------------------------------------- */
model_s::model_s( int max_s, int max_c, int max_o, int c_lim ) :
// Copy settings into the model:
max_symbol(max_s),
max_context(max_c),
max_order(max_o + 1),
max_count(c_lim),
current_order(max_o + 1),
sb0_count(max_s),
totals(max_s + 2),
scoreboard(new bool[max_s]),
contexts(max_o + 3)
{
std::fill(scoreboard, scoreboard + max_symbol, false);
// set up null table
table_s* null_table = new table_s;
null_table->counts = std::vector<uint16_t>(max_symbol, uint16_t(1)); // Set all probabilities to 1.
// set up internal counts
null_table->max_count = 1;
null_table->max_symbol = max_symbol;
// set up start table
table_s* start_table = new table_s;
start_table->links = std::vector<table_s*>(max_context);
// integrate tables into contexts
contexts[ 0 ] = null_table;
contexts[ 1 ] = start_table;
// build initial 'normal' tables
for (int i = 2; i <= max_order; i++ ) {
// set up current order table
contexts[i] = new table_s;
// build forward links
if ( i < max_order ) {
contexts[i]->links = std::vector<table_s*>(max_context);
}
contexts[ i - 1 ]->links[ 0 ] = contexts[ i ];
}
}
/* -----------------------------------------------
model class destructor - recursive cleanup of memory is done here
----------------------------------------------- */
model_s::~model_s()
{
// clean up each 'normal' table
delete contexts[1];
// clean up null table
delete contexts[0];
// free everything else
delete[] scoreboard;
}
/* -----------------------------------------------
Updates statistics for a specific symbol / resets to highest order.
Use -1 if you just want to reset without updating statistics.
----------------------------------------------- */
void model_s::update_model( int symbol )
{
// only contexts, that were actually used to encode
// the symbol get its count updated
if ( symbol >= 0 ) {
for (int local_order = ( current_order < 1 ) ? 1 : current_order;
local_order <= max_order; local_order++ ) {
table_s* context = contexts[ local_order ];
auto& count = context->counts[symbol];
// update count for specific symbol & scale
count++;
// store side information for totalize_table
context->max_count = std::max(count, context->max_count);
context->max_symbol = std::max(uint16_t(symbol + 1), context->max_symbol);
// if count for that symbol have gone above the maximum count
// the table has to be resized (scale factor 2)
if (count == max_count) {
context->rescale_table();
}
}
}
// reset scoreboard and current order
current_order = max_order;
std::fill(scoreboard, scoreboard + max_symbol, false);
sb0_count = max_symbol;
}
/* -----------------------------------------------
shift in one context (max no of contexts is max_c)
----------------------------------------------- */
void model_s::shift_context( int c )
{
// shifting is not possible if max_order is below 1
// or context index is negative
if ( ( max_order < 2 ) || ( c < 0 ) ) return;
// shift each orders' context
for (int i = max_order; i > 1; i-- ) {
// this is the new current order context
table_s* context = contexts[ i - 1 ]->links[ c ];
// check if context exists, build if needed
if ( context == nullptr ) {
// reserve memory for next table_s
context = new table_s;
// finished here if this is a max order context
if ( i < max_order ) {
// build links to higher order tables otherwise
context->links.resize(max_context);
}
// put context to its right place
contexts[ i - 1 ]->links[ c ] = context;
}
// switch context
contexts[ i ] = context;
}
}
/* -----------------------------------------------
Flushes the entire model by calling rescale_table on all contexts.
----------------------------------------------- */
void model_s::flush_model()
{
contexts[1]->recursive_flush();
}
/* -----------------------------------------------
Excludes every symbol above c.
----------------------------------------------- */
void model_s::exclude_symbols(int c)
{
// exclusions are back to normal after update_model is used
for ( c = c + 1; c < max_symbol; c++ ) {
if ( !scoreboard[ c ] ) {
scoreboard[ c ] = true;
sb0_count--;
}
}
}
/* -----------------------------------------------
converts an int to a symbol, needed only when encoding
----------------------------------------------- */
int model_s::convert_int_to_symbol( int c, symbol *s )
{
// search the symbol c in the current context table_s,
// return scale, low- and high counts
// totalize table for the current context
table_s* context = contexts[ current_order ];
totalize_table( context );
// finding the scale is easy
s->scale = totals[ 0 ];
// check if that symbol exists in the current table. send escape otherwise
if ( context->counts[ c ] > 0 ) {
// return high and low count for the current symbol
s->low_count = totals[ c + 2 ];
s->high_count = totals[ c + 1 ];
return 0;
}
// return high and low count for the escape symbol
s->low_count = totals[ 1 ];
s->high_count = totals[ 0 ];
current_order--;
return 1;
}
/* -----------------------------------------------
returns the current context scale needed only when decoding
----------------------------------------------- */
void model_s::get_symbol_scale( symbol *s )
{
// getting the scale is easy: totalize the table_s, use accumulated count -> done
totalize_table( contexts[ current_order ] );
s->scale = totals[ 0 ];
}
/* -----------------------------------------------
converts a count to an int, called after get_symbol_scale
----------------------------------------------- */
int model_s::convert_symbol_to_int(uint32_t count, symbol *s)
{
// seek the symbol that matches the count,
// also, set low- and high count for the symbol - it has to be removed from the stream
// go through the totals table, search the symbol that matches the count
uint32_t c;
for (c = 1; c < totals.size(); c++) {
if (count >= totals[c]) {
break;
}
}
// set up the current symbol
s->low_count = totals[c]; // It is guaranteed that there exists such a symbol.
s->high_count = totals[c - 1]; // This is guaranteed to not go out of bounds since the search started at index 1 of totals.
// send escape if escape symbol encountered
if (c == 1) {
current_order--;
return ESCAPE_SYMBOL;
}
// return symbol value
return c - 2 ; // Since c is not one and is a positive number, this will be nonnegative.
}
/* -----------------------------------------------
totals are calculated by accumulating counts in the current table_s
----------------------------------------------- */
void model_s::totalize_table( table_s* context )
{
// update exclusion is used, so this has to be done each time
// escape probability calculation also takes place here
// accumulated counts must never exceed CODER_MAXSCALE
// as CODER_MAXSCALE is big enough, though, (2^29), this shouldn't happen and is not checked
const auto& counts = context->counts;
// check counts
if (!counts.empty()) { // if counts are already set
// locally store current fill/symbol count
int local_symb = sb0_count;
// set the last symbol of the totals to zero
int i = context->max_symbol - 1;
totals[i + 2] = 0;
// (re)set current total
uint32_t curr_total = 0;
// go reverse though the whole counts table and accumulate counts
// leave space at the beginning of the table for the escape symbol
for (; i >= 0; i--) {
// only count probability if the current symbol is not 'scoreboard - excluded'
if (!scoreboard[i]) {
uint16_t curr_count = counts[i];
if (curr_count > 0) {
// add counts for the current symbol
curr_total += curr_count;
// exclude symbol from scoreboard
scoreboard[i] = true;
sb0_count--;
}
}
totals[i + 1] = curr_total;
}
// here the escape calculation needs to take place
uint32_t esc_prob;
if (local_symb == sb0_count) {
esc_prob = 1;
} else if (sb0_count == 0) {
esc_prob = 0;
} else {
// esc_prob = 1;
esc_prob = sb0_count * ( local_symb - sb0_count );
esc_prob /= ( local_symb * context->max_count );
esc_prob++;
}
// include escape probability in totals table
totals[ 0 ] = totals[ 1 ] + esc_prob;
} else { // if counts are not already set
// setup counts for current table
context->counts.resize(max_symbol);
// set totals table -> only escape probability included
totals[ 0 ] = 1;
totals[ 1 ] = 0;
}
}
/* -----------------------------------------------
special version of model_s for binary coding
boundaries of this model:
... (maximum symbol) -> 2 (0 or 1 )
max_c (maximum context) -> 1 <= max_c <= 1024 (???)
max_o (maximum order) -> -1 <= max_o <= 4
----------------------------------------------- */
model_b::model_b( int max_c, int max_o, int c_lim ) :
// Copy settings into the model:
max_context(max_c),
max_order(max_o + 1),
max_count(c_lim),
contexts(max_o + 3)
{
// set up null table
table* null_table = new table;
null_table->counts = std::vector<uint16_t>(2, uint16_t(1));
null_table->scale = uint32_t(2);
// set up start table
table* start_table = new table;
start_table->links = std::vector<table*>(max_context);
// integrate tables into contexts
contexts[ 0 ] = null_table;
contexts[ 1 ] = start_table;
// build initial 'normal' tables
for (int i = 2; i <= max_order; i++ ) {
// set up current order table
contexts[i] = new table;
// build forward links
if ( i < max_order ) {
contexts[i]->links = std::vector<table*>(max_context);
}
contexts[ i - 1 ]->links[ 0 ] = contexts[ i ];
}
}
/* -----------------------------------------------
model class destructor - recursive cleanup of memory is done here
----------------------------------------------- */
model_b::~model_b()
{
// clean up each 'normal' table
delete contexts[1];
// clean up null table
delete contexts[0];
}
/* -----------------------------------------------
updates statistics for a specific symbol / resets to highest order
----------------------------------------------- */
void model_b::update_model( int symbol )
{
// use -1 if you just want to reset without updating statistics
table* context = contexts[ max_order ];
// only contexts, that were actually used to encode
// the symbol get their counts updated
if ( ( symbol >= 0 ) && ( max_order >= 0 ) ) {
// update count for specific symbol & scale
context->counts[ symbol ]++;
context->scale++;
// if counts for that symbol have gone above the maximum count
// the table has to be resized (scale factor 2)
if ( context->counts[ symbol ] >= max_count )
context->rescale_table();
}
}
/* -----------------------------------------------
shift in one context (max no of contexts is max_c)
----------------------------------------------- */
void model_b::shift_context( int c )
{
// shifting is not possible if max_order is below 1
// or context index is negative
if ( (max_order < 2 ) || ( c < 0 ) ) return;
// shift each orders' context
for (int i = max_order; i > 1; i-- ) {
// this is the new current order context
table* context = contexts[ i - 1 ]->links[ c ];
// check if context exists, build if needed
if ( context == nullptr ) {
// reserve memory for next table
context = new table;
// finished here if this is a max order context
if ( i < max_order) {
// build links to higher order tables otherwise
context->links.resize(max_context);
}
// put context to its right place
contexts[ i - 1 ]->links[ c ] = context;
}
// switch context
contexts[ i ] = context;
}
}
/* -----------------------------------------------
Flushes the entire model by calling rescale_table on all contexts.
----------------------------------------------- */
void model_b::flush_model()
{
contexts[1]->recursive_flush();
}
/* -----------------------------------------------
converts an int to a symbol, needed only when encoding
----------------------------------------------- */
int model_b::convert_int_to_symbol( int c, symbol *s )
{
table* context = contexts[ max_order ];
// check if counts are available
context->check_counts();
// finding the scale is easy
s->scale = context->scale;
// return high and low count for current symbol
if ( c == 0 ) { // if 0 is to be encoded
s->low_count = uint32_t(0);
s->high_count = context->counts[ 0 ];
}
else { // if 1 is to be encoded
s->low_count = context->counts[ 0 ];
s->high_count = context->scale;
}
return 1;
}
/* -----------------------------------------------
returns the current context scale needed only when decoding
----------------------------------------------- */
void model_b::get_symbol_scale( symbol *s )
{
table* context = contexts[ max_order ];
// check if counts are available
context->check_counts();
// getting the scale is easy
s->scale = context->scale;
}
/* -----------------------------------------------
converts a count to an int, called after get_symbol_scale
----------------------------------------------- */
int model_b::convert_symbol_to_int(uint32_t count, symbol *s)
{
table* context = contexts[ max_order ];
auto counts0 = context->counts[ 0 ];
// set up the current symbol
if ( count < counts0 ) {
s->low_count = uint32_t(0);
s->high_count = counts0;
return 0;
}
else {
s->low_count = counts0;
s->high_count = s->scale;
return 1;
}
}
software/photo_compress_archiver/packJPG/aricoder.h
#ifndef ARICODER_H
#define ARICODER_H
#include <cstdint>
#include "bitops.h"
#include <vector>
#include <algorithm>
// defines for coder
constexpr uint32_t CODER_USE_BITS = 31; // Must never be above 31.
constexpr uint32_t CODER_LIMIT100 = uint32_t(1 << CODER_USE_BITS);
constexpr uint32_t CODER_LIMIT025 = CODER_LIMIT100 / 4;
constexpr uint32_t CODER_LIMIT050 = (CODER_LIMIT100 / 4) * 2;
constexpr uint32_t CODER_LIMIT075 = (CODER_LIMIT100 / 4) * 3;
constexpr uint32_t CODER_MAXSCALE = CODER_LIMIT025 - 1;
constexpr uint32_t ESCAPE_SYMBOL = CODER_LIMIT025;
// symbol struct, used in arithmetic coding
struct symbol {
uint32_t low_count;
uint32_t high_count;
uint32_t scale;
};
// table struct, used in in statistical models,
// holding all info needed for one context
struct table {
// counts for each symbol contained in the table
std::vector<uint16_t> counts;
// links to higher order contexts
std::vector<table*> links;
// accumulated counts
uint32_t scale = uint32_t(0);
/* -----------------------------------------------
Recursively deletes all the tables pointed to in links.
----------------------------------------------- */
~table() {
for (auto& link : links) {
if (link != nullptr) {
delete link;
}
}
}
/* -----------------------------------------------
Checks if counts exist, creating it if it does not.
----------------------------------------------- */
inline void check_counts() {
// check if counts are available
if (counts.empty()) {
// setup counts for current table
counts.resize(2, uint16_t(1));
// set scale
scale = uint32_t(2);
}
}
/* -----------------------------------------------
Resizes the table by rightshifting each count by 1.
----------------------------------------------- */
inline void rescale_table() {
// Do nothing if counts is not set:
if (!counts.empty()) {
// Scale the table by bitshifting each count, be careful not to set any count zero:
counts[0] = std::max(uint16_t(1), uint16_t(counts[0] >> 1));
counts[1] = std::max(uint16_t(1), uint16_t(counts[1] >> 1));
scale = counts[0] + counts[1];
}
}
/* -----------------------------------------------
Recursively runs rescale_table on this and all linked contexts.
----------------------------------------------- */
inline void recursive_flush() {
for (auto& link : links) {
if (link != nullptr) {
link->recursive_flush();
}
}
// rescale specific table
rescale_table();
}
};
// special table struct, used in in model_s,
// holding additional info for a speedier 'totalize_table'
struct table_s {
// counts for each symbol contained in the table
std::vector<uint16_t> counts;
// links to higher order contexts
std::vector<table_s*> links;
// speedup info
uint16_t max_count = uint16_t(0);
uint16_t max_symbol = uint16_t(0);
/* -----------------------------------------------
Recursively deletes all the tables pointed to in links.
----------------------------------------------- */
~table_s() {
for (auto& link : links) {
if (link != nullptr) {
delete link;
}
}
}
/* -----------------------------------------------
Resizes the table by rightshifting each count by 1.
----------------------------------------------- */
inline void rescale_table() {
// Nothing to do if counts has not been set.
if (counts.empty()) return;
// now scale the table by bitshifting each count
int lst_symbol = max_symbol;
int i;
for (i = 0; i < lst_symbol; i++) {
counts[i] >>= 1; // Counts will not become negative since it is an unsigned type.
}
// also rescale tables max count
max_count >>= 1;
// seek for new last symbol
for (i = lst_symbol - 1; i >= 0; i--) {
if (counts[i] > 0) {
break;
}
}
max_symbol = i + 1;
}
/* -----------------------------------------------
Recursively runs rescale_table on this and all linked contexts.
----------------------------------------------- */
inline void recursive_flush() {
for (auto& link : links) {
if (link != nullptr) {
link->recursive_flush();
}
}
// rescale specific table
rescale_table();
}
};
/* -----------------------------------------------
class for arithmetic coding of data to/from iostream
----------------------------------------------- */
class aricoder
{
public:
aricoder( iostream* stream, StreamMode iomode );
~aricoder();
void encode( symbol* s );
unsigned int decode_count( symbol* s );
void decode( symbol* s );
private:
template<uint8_t bit>
void write_bit() {
// add bit at last position
bbyte = (bbyte << 1) | bit;
// increment bit position
cbit++;
// write bit if done
if (cbit == 8) {
sptr->write_byte(bbyte);
cbit = 0;
}
}
void writeNrbitsAsZero();
void writeNrbitsAsOne();
unsigned char read_bit();
// i/o variables
iostream* sptr; // Pointer to iostream for reading/writing.
const StreamMode mode;
unsigned char bbyte = 0;
unsigned char cbit = 0;
// arithmetic coding variables
unsigned int ccode = 0;
unsigned int clow = 0;
unsigned int chigh = CODER_LIMIT100 - 1;
unsigned int cstep = 0;
unsigned int nrbits = 0;
};
/* -----------------------------------------------
universal statistical model for arithmetic coding
----------------------------------------------- */
class model_s
{
public:
model_s( int max_s, int max_c, int max_o, int c_lim );
~model_s();
void update_model( int symbol );
void shift_context( int c );
void flush_model();
void exclude_symbols(int c);
int convert_int_to_symbol( int c, symbol *s );
void get_symbol_scale( symbol *s );
int convert_symbol_to_int(uint32_t count, symbol *s);
private:
inline void totalize_table(table_s* context);
const int max_symbol;
const int max_context;
const int max_order;
const int max_count;
int current_order;
int sb0_count;
std::vector<uint32_t> totals;
bool* scoreboard;
std::vector<table_s*> contexts;
};
/* -----------------------------------------------
binary statistical model for arithmetic coding
----------------------------------------------- */
class model_b
{
public:
model_b( int max_c, int max_o, int c_lim );
~model_b();
void update_model( int symbol );
void shift_context( int c );
void flush_model();
int convert_int_to_symbol( int c, symbol *s );
void get_symbol_scale( symbol *s );
int convert_symbol_to_int(uint32_t count, symbol *s);
private:
const int max_context;
const int max_order;
const int max_count;
std::vector<table*> contexts;
};
// Base case for shifting an arbitrary number of contexts into the model.
template <typename M>
static void shift_model(M) {}
// Shift an arbitrary number of contexts into the model (at most max_c contexts).
template <typename M, typename C, typename... Cargs>
static void shift_model(M model, C context, Cargs ... contextList) {
model->shift_context(context);
shift_model(model, contextList...);
}
/* -----------------------------------------------
generic model_s encoder function
----------------------------------------------- */
static inline void encode_ari( aricoder* encoder, model_s* model, int c )
{
symbol s;
int esc;
do {
esc = model->convert_int_to_symbol( c, &s );
encoder->encode( &s );
} while ( esc );
model->update_model( c );
}
/* -----------------------------------------------
generic model_s decoder function
----------------------------------------------- */
static inline int decode_ari( aricoder* decoder, model_s* model )
{
symbol s;
uint32_t count;
int c;
do{
model->get_symbol_scale( &s );
count = decoder->decode_count( &s );
c = model->convert_symbol_to_int( count, &s );
decoder->decode( &s );
} while ( c == ESCAPE_SYMBOL );
model->update_model( c );
return c;
}
/* -----------------------------------------------
generic model_b encoder function
----------------------------------------------- */
static inline void encode_ari( aricoder* encoder, model_b* model, int c )
{
symbol s;
model->convert_int_to_symbol( c, &s );
encoder->encode( &s );
model->update_model( c );
}
/* -----------------------------------------------
generic model_b decoder function
----------------------------------------------- */
static inline int decode_ari( aricoder* decoder, model_b* model )
{
symbol s;
model->get_symbol_scale( &s );
uint32_t count = decoder->decode_count( &s );
int c = model->convert_symbol_to_int( count, &s );
decoder->decode( &s );
model->update_model( c );
return c;
}
#endif
software/photo_compress_archiver/packJPG/bitops.cpp
/*
This file contains special classes for bitwise
reading and writing of arrays
*/
#include "bitops.h"
#include <algorithm>
#include <array>
#include <stdio.h>
#include <stdlib.h>
#include <vector>
#if defined(_WIN32) || defined(WIN32)
#include <fcntl.h>
#include <io.h>
#endif
// special realloc with guaranteed free() of previous memory
template <class T>
static inline T* frealloc( T* ptr, size_t size ) {
T* n_ptr = (T*)realloc( ptr, (size) ? size : 1 );
if ( n_ptr == nullptr ) free( ptr );
return n_ptr;
}
/* -----------------------------------------------
constructor for abitreader class
----------------------------------------------- */
abitreader::abitreader( unsigned char* array, int size )
{
cbyte = 0;
cbit = 8;
peof_ = 0;
eof_ = false;
data = array;
lbyte = size;
}
/* -----------------------------------------------
destructor for abitreader class
----------------------------------------------- */
... This diff was truncated because it exceeds the maximum size that can be displayed.

Also available in: Unified diff