Friday, December 1, 2017

Freezer

https://github.com/arqex/freezer
Immutable.js alternative

Datomic Database

http://www.datomic.com/
Built with Clojure
Immutable: you can get a view of the DB on any past moment.
Pluggable persistence: PostgreSQL, Cassandra, ...

Wednesday, November 8, 2017

LLVM to JavaScript compilers

emscripten compiler
http://kripken.github.io/emscripten-site/
C/C++ to JavaScript (asm.js)

em-DOSbox
DOSBox compiled with emscripten

Windows 95 in your browser
Windows95 installed on em-DOSBox

Windows 2000 & Linux
https://bellard.org/jslinux/

Monday, September 4, 2017

Removing a VM on ESXi standalone with Ansible

The pysphere Python package must be installed.

 hosts: 127.0.0.1
  connection: local
  user: username
  sudo: false
  gather_facts: false
  serial: 1
  
  tasks:

    - vsphere_guest: 
        vcenter_hostname: <<ESXi-IP>> 
        username: root 
        password: <<password>>
        esxi: 
          datacenter: ha-datacenter
        guest: <<virtual-machine-name>>
        state: absent
        force: yes
        validate_certs: false

Monday, August 28, 2017

Spring boot conditional configuration

SpEL Ternary Operator

The @Value annotation allow setting SpEL expressions: 
@Value("#{httpMode.equals(\"secure\") ? 43 : 80}")
int portNumber;

Profiles

Profiles are used to alter the default bean selection and wiring and also de Spring environment properties source.

Activation

Any number of profiles can be activated.

Profiles can be activated by setting the spring.profiles.active Spring environment variable (in a property file or any other property source).
spring.profiles.active=profile1,profile2,...

The  --spring.profiles.active=profile1,profile2,... command line switch can also be used.

This methods allow to unconditionally set the full active profile set but you can also add (not replace) additional profiles to the existing ones with the spring.profiles.include property.

Profiles can also be activated programatically with SpringApplication.setAdditionalProfiles or the ConfigurableEnvironment interface.

Profiles can also be activated on tests with the @ActiveProfiles annotation.

Default profile

When no profile is activated there is still a "default" profile active.

The default profile can also be activated with other profiles by explicitly adding it ("default") to the active profile set.

@Profile annotation


@Profile({"profile1", "profile2", ...})
Loads the annotated bean when one of the profiles of the annotation is active.

It can be used on any @Component, complete @Configuration or any of its @Bean methods.

@Profile({"!profile1"})
Loads the bean when the profile is not present.

If the @Profile annotation is omitted, registration will occur regardless of which (if any) profiles are active.

@Profile("default", "profile1", ...)
Any bean annotated with @Profile will not be loaded when no profile is set unless the "default" profile is included in the annotation.

@IfProfileActive

https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/test/annotation/IfProfileValue.html

Is an old Spring 2.0 annotation that enables or disables a test based on a property or set of Spring environment property value. It has no relation with @Profile.

@IfProfileValue(name = "java.vendor", value = "Oracle Corporation")
public void testSomething() {
   ...
}

@Primary annotation

https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/context/annotation/Primary.html

Indicates that a bean should be given preference when multiple candidates are qualified to autowire a single-valued dependency.

It can be used on any @Component, complete @Configuration or any of its @Bean methods.

Can be combined with @Profile providing a way to replace any bean with the profile ones.

@Conditional annotation


The most powerfull annotation for conditional configuration. Spring Boot module makes heavy use of it.

It can be used on any @Component, complete @Configuration or any of its @Bean methods.


References

Saturday, August 19, 2017

Asynchronous and Parallel Programming in C#

History and current status

  • .NET Framework 1.1 - C# 1.1: Asynchronous Programming Model (APM).
  • .NET Framework 2 - C# 2: Event Asynchronous Pattern (EAP). 
  • .NET Framework 4 - C# 4: Task Parallel Library (TPL). Mostly multi-threading oriented.
  • .NET Framework 4.5 - C# 5: Asynchronous methods (async/await). Task Continuations are often not needed. Huge code noise reduction. Better support in the Basic Class Library (BCL). "Async" suffix in many methods. Mostly asynchronous programming oriented.
Today the preferred way to handle asynchronous programming is by using Tasks with async/await first and then TPL.

Caution! in the TPL world tasks are usually "Delegate tasks" (tied to threads) but async/await tasks are mostly "Promise tasks". See https://blog.stephencleary.com/2014/04/a-tour-of-task-part-0-overview.html

Extensions

Resources

Thursday, August 10, 2017

MobaXterm


Vagrant
Vagrant does not run when the env variable VAGRANT_DETECTED_OD=cygwin. It can be solved with:
export VAGRANT_DETECTED_OS=

Thursday, August 3, 2017

Thursday, July 20, 2017

Open Source BI and Visualization Web Applications

Superset

https://github.com/apache/incubator-superset
From Airbnb, now Apache

Redash

https://redash.io/

Metabase

http://www.metabase.com/

Zepellin

https://zeppelin.apache.org/

Blazer

https://github.com/ankane/blazer

Kibana

https://www.elastic.co/products/kibana


Wednesday, June 28, 2017

Container Ochestrators

https://netsil.com/blog/kubernetes-vs-docker-vs-mesosphere/

Swarm


Kubernetes

Kubernetes is an open source container orchestration framework (create and manage multiple docker containers).
  • By Google.
  • Huge GitHub activity.
  • Runs everywhere (on-premise clusters or on cloud with support for AWS,GCE and Azure out of the box).
  • Declarative
  • REST API
  • Builds the cluster and maintains it: network abstraction and management, scale adding nodes, remove nodes, etc.
  • Service upgrade and rollback.
  • Resource abstraction and management: frees you from the machine. It does the segregation of the services in the machines. You only define the resource needs of each service in CPU, GB or percentages.
  • Storage abstraction and management: docker volumes, upgrades/restarts, 

Spark support

DC/OS 

  • Mesos based.
  • Container technology agnostic but now centered on Docker.
  • Non containerized applications can also be deployed.

Thursday, June 1, 2017

Kafka Connect

Standalone 
bin/connect-standalone worker.properties connector1.properties connector2.properties ...

Converters
Connectors
http://www.landoop.com/kafka/connectors/

Tasks
A single call to "put" may include messages from different topics.
"flush" is invoked periodically (offset.flush.interval.ms).
Both put and flush may be invoked with zero messages.

Custom connector
https://github.com/mcandre/hello-kafka-connect

Tuesday, May 16, 2017

HP Vertica

  • HP columnar database. 
  • PostgreSQL fork.
  • Updates and inserts (even batched) are slow. Bulk loading required.
Tools
  • Binaries at opt/vertica/bin and opt/vertica/bin
vsql
adminTools

VM credentials
dbadmin/password
root/password

Example Database
Scripts location: /opt/vertica/examples
Installation: ./install_example VMart 
Verification: launch vsql and select count(1) from store.store_sales_fact;

Singularities
Bulk loading
Tips
https://github.com/jackghm/Vertica/wiki/HP-Vertica-Tips,-Tricks,-and-Best-Practices

Monday, May 8, 2017

Notes on Apache Kafka


http://davewentzel.com/content/kafka-notes/

Clustering
Based on Zookeeper
https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

Log compaction
Cosumer receives at least the last message of each key.
http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/

JSON serialization
org.apache.kafka.connect.json.JsonConverter
Easier to setup
https://blog.knoldus.com/2017/01/30/kafka-sending-object-as-a-message/

AVRO serialization
io.confluent.connect.avro.AvroConverter
Recommended: schema based, fast, compact, versioning (evolution)
Schema registry needed (Confluent distro)
Schemas are automatically registered by the producers
http://cloudurable.com/blog/avro/index.html
http://cloudurable.com/blog/kafka-avro-schema-registry/index.html
https://www.slideshare.net/JeanPaulAzar1/kafka-and-avro-with-confluent-schema-registry

Compression
GZIP or Snappy
Less bandwitdh and disk space, more CPU resources.

Topics and Partitions
Multiple topics per producer: https://stackoverflow.com/questions/21376715/how-many-producers-to-create-in-kafka

Distributions
Producer
http://cloudurable.com/blog/kafka-tutorial-kafka-producer-advanced-java-examples/index.html

Friday, March 17, 2017

Angular CLI 1.0.0-RC.1

Angular: 2.4.0
Material: 2.0.0-beta.2

npm i -g @angular/cli@latest
ng new myapp --skip-git --style=scss --routing
npm install --save @angular/material
npm install --save hammerjs

Thursday, February 9, 2017

Kryo serialization

≃ 1/4 smaller results and more than 3x faster than standard Java.
Widely used (Spark, ...).
Kryo instances  are not thread safe.

Serialization

Kryo kryo = new Kryo();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
Output output = new Output(bos);
byte bytes[]; 
try {
  kryo.writeClassAndObject(output, obj);
  bytes = output.toBytes();
} finally {
  bos.close();
}

Deserialization

Kryo kryo = new Kryo();
Input input = new Input(bytes);
Object obj;
try {
  obj = kryo.readClassAndObject(input);
}
finally {
  input.close();
}

Sunday, February 5, 2017

PostgreSQL tuning

About PostgreSQL

Periodic maintenance

Conventional (disk based)


max_connections
Number of planned connections.

effective_cache_size
Expected memory to be available in the OS and PostgreSQL buffer caches, not an allocation!. Used only by the PostgreSQL query planner to figure out whether plans it's considering would be expected to fit in RAM or not.
Setting effective_cache_size to 1/2 of total memory would be a normal conservative setting, and 3/4 of memory is a more aggressive but still reasonable amount.

shared_buffers
Page caching.
25% of system RAM and no more than 8GB.

Alternative?: higher values may perform much better if you can comfortably fit the working set inside shared_buffers leaving a generous amount of memory left over for other purposes. http://rhaas.blogspot.co.at/2012/03/tuning-sharedbuffers-and-walbuffers.html 

work_mem
Intermediante results, sort, hashjoins, materialized views, etc.
4 to 64MB. 

maintenance_work_mem
Maintenance operations like vacuuming and index creation.
5% of system RAM and no more than 512MB. 

wal_buffers
No more than 16MB. 

checkpoint_segments
Deprecated on 9.5. Now max_wal_size.
Better performance on bulk data loads. 

checkpoint_completion_target
0.9 will decrease the performance impact of checkpointing on a busy system.

checkpoint_timeout
Increasing checkpoint_timeout from 5 minutes to a larger value, such as 15 minutes, can reduce the I/O load on your system, especially when using large values for shared_buffers.

random_page_cost and seq_page_cost
Planner parameters.
An estimate for the relative cost of disk seeks.
2 and 1 (0.1 for both values if the DB completely fits in memory).

huge_pages

In-memory